[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:04 dist-test-slave-hvv5 krb5kdc[1684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:04 dist-test-slave-hvv5 krb5kdc[1684](info): set up 2 sockets
Sep 25 15:25:04 dist-test-slave-hvv5 krb5kdc[1684](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:25:05.101365 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:05.241554 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:35987
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:05.326556 1697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:05.326833 1697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:05.326874 1697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:05.328104 1697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:05.328181 1697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:05.328207 1697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:05.328233 1697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:05.328266 1697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:05.329944 1697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:05.330317 1697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:05.330590 1697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:05.336851 1703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.336958 1705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.336869 1702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:05.337109 1697 server_base.cc:1034] running on GCE node
I20260925 15:25:05.337374 1697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:05.337698 1697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:05.338910 1697 hybrid_clock.cc:648] HybridClock initialized: now 1790349905338898 us; error 36 us; skew 500 ppm
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:05.341614 1697 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:05.342098 1697 webserver.cc:460] Webserver started at http://127.1.163.126:42773/ using document root <none> and password file <none>
I20260925 15:25:05.342409 1697 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:05.342461 1697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:05.342680 1697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:05.343791 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "528297c3b4f64fb794a641a2898e8c3b"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "1581ce04b7c139952e6c13daea38823e"
server_key_iv: "a4481bff5822a77400d3ab5d37acb987"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.344174 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "528297c3b4f64fb794a641a2898e8c3b"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "1581ce04b7c139952e6c13daea38823e"
server_key_iv: "a4481bff5822a77400d3ab5d37acb987"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.345965 1697 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:25:05.346868 1712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.347087 1697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:05.347184 1697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "528297c3b4f64fb794a641a2898e8c3b"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "1581ce04b7c139952e6c13daea38823e"
server_key_iv: "a4481bff5822a77400d3ab5d37acb987"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.347266 1697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:05.371794 1697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:05.374174 1697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:05.374334 1697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:05.378654 1697 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:35987
I20260925 15:25:05.378688 1764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:35987 every 8 connection(s)
I20260925 15:25:05.379069 1697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:05.379942 1765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:05.382486 1765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b: Bootstrap starting.
I20260925 15:25:05.383154 1765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:05.383509 1765 log.cc:824] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:05.384299 1765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b: No bootstrap required, opened a new log
I20260925 15:25:05.385969 1765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } }
I20260925 15:25:05.386138 1765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:05.386163 1765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528297c3b4f64fb794a641a2898e8c3b, State: Initialized, Role: FOLLOWER
I20260925 15:25:05.386338 1765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } }
I20260925 15:25:05.386399 1765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:05.386458 1765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:05.386507 1765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:05.387220 1765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } }
I20260925 15:25:05.387220 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 1697
I20260925 15:25:05.387324 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:05.387336 1765 leader_election.cc:302] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528297c3b4f64fb794a641a2898e8c3b; no voters:
I20260925 15:25:05.387516 1765 leader_election.cc:288] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:05.387573 1768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:05.387745 1768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [term 1 LEADER]: Becoming Leader. State: Replica: 528297c3b4f64fb794a641a2898e8c3b, State: Running, Role: LEADER
I20260925 15:25:05.387773 1765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:05.387842 1768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } }
I20260925 15:25:05.388146 1768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528297c3b4f64fb794a641a2898e8c3b. Latest consensus state: current_term: 1 leader_uuid: "528297c3b4f64fb794a641a2898e8c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } } }
I20260925 15:25:05.388181 1677 external_mini_cluster.cc:1428] Setting key 3fabe42e9deb13bf044639f0c012a814
I20260925 15:25:05.388224 1768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:05.388307 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528297c3b4f64fb794a641a2898e8c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528297c3b4f64fb794a641a2898e8c3b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35987 } } }
I20260925 15:25:05.388360 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:05.389331 1773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:05.389993 1773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:05.391497 1773 catalog_manager.cc:1347] Generated new cluster ID: 2e7385a140c84eb8bdd868e1a93d05a0
I20260925 15:25:05.391551 1773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:05.405599 1773 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:05.406418 1773 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:05.409860 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.389503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.1.163.126:35987)
0925 15:25:05.389760 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.389771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.390405 (+ 634us) server_negotiation.cc:407] Connection header received
0925 15:25:05.392964 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.392972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.393036 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.393163 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.395384 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.397129 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.398791 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.399433 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.399545 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.404732 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.404737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.404738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.404753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.407141 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.407714 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.407718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.407719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.407851 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.408131 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.408132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.408132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.408416 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.408585 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.409436 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.409607 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
I20260925 15:25:05.411921 1773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528297c3b4f64fb794a641a2898e8c3b: Generated new TSK 0
I20260925 15:25:05.412118 1773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:05.444217 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:05.535645 1791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:05.535878 1791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:05.535907 1791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:05.537104 1791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:05.537146 1791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:05.537174 1791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:05.538647 1791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:05.538918 1791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:05.539122 1791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:05.544399 1796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.544421 1799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.544407 1797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:05.544469 1791 server_base.cc:1034] running on GCE node
I20260925 15:25:05.544919 1791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:05.545188 1791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:05.546394 1791 hybrid_clock.cc:648] HybridClock initialized: now 1790349905546387 us; error 38 us; skew 500 ppm
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:05.549187 1791 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:05.549685 1791 webserver.cc:460] Webserver started at http://127.1.163.65:42425/ using document root <none> and password file <none>
I20260925 15:25:05.549923 1791 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:05.550005 1791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:05.550153 1791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:05.551283 1791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "707674fd727b4d9eb68365227522d61f"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "7941d957ba62a34e2d7b332353ae3067"
server_key_iv: "1ba8047b946a1a15a7da348afbd1702b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.551604 1791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "707674fd727b4d9eb68365227522d61f"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "7941d957ba62a34e2d7b332353ae3067"
server_key_iv: "1ba8047b946a1a15a7da348afbd1702b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.553112 1791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:05.553820 1806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.553995 1791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:05.554100 1791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "707674fd727b4d9eb68365227522d61f"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "7941d957ba62a34e2d7b332353ae3067"
server_key_iv: "1ba8047b946a1a15a7da348afbd1702b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.554177 1791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:05.576335 1791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:05.578981 1791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:05.579156 1791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:05.579417 1791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:05.579852 1791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:05.579916 1791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.579964 1791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:05.579988 1791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.586058 1791 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:42501
I20260925 15:25:05.586102 1919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:42501 every 8 connection(s)
I20260925 15:25:05.586428 1791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:05.589718 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 1791
I20260925 15:25:05.589810 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:05.590055 1677 external_mini_cluster.cc:1428] Setting key 536bf37d904889640751190979841a4d
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:05.597576 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.587263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:59273 (local address 127.1.163.126:35987)
0925 15:25:05.587313 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.587316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.587547 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:25:05.588112 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.588114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.588126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.588184 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.589441 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.590134 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.591296 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.591529 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.591615 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.594238 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.594240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.594241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.594256 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.596057 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.596483 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.596485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.596485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.596578 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.596831 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.596832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.596832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.596993 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.597075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.597335 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.597456 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:25:05.597606 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.587218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35987 (local address 127.1.163.65:59273)
0925 15:25:05.587416 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:05.587454 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:05.587997 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:05.588260 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:05.588265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:05.589115 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:05.589323 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.589328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.590221 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.590222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.591205 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.591208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.591627 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.591629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.591771 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.592349 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:05.592357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:05.594100 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:05.596149 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.596156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.596158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.596402 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.596651 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.596652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.596652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.596760 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.597065 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:05.597070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:05.597197 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 15:25:05.597284 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:25:05.597417 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260925 15:25:05.598156 1920 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35987
I20260925 15:25:05.598297 1920 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:05.598490 1920 heartbeater.cc:511] Master 127.1.163.126:35987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:05.599398 1729 ts_manager.cc:194] Registered new tserver with Master: 707674fd727b4d9eb68365227522d61f (127.1.163.65:42501)
I20260925 15:25:05.600052 1729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:59273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:05.620246 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:05.707083 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:05.707348 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:05.707391 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:05.708714 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:05.708776 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:05.708819 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:05.710592 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:05.710918 1925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:05.711185 1925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:05.716266 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.716312 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.716266 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:05.716632 1925 server_base.cc:1034] running on GCE node
I20260925 15:25:05.716864 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:05.717083 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:05.718237 1925 hybrid_clock.cc:648] HybridClock initialized: now 1790349905718216 us; error 37 us; skew 500 ppm
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:05.721102 1925 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:05.721587 1925 webserver.cc:460] Webserver started at http://127.1.163.66:46205/ using document root <none> and password file <none>
I20260925 15:25:05.721853 1925 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:05.721902 1925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:05.722075 1925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:05.723270 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "40e152f495fe46a3b5e9f353e95c3969"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "c9468cd0e4a4ac66958336b83193ff91"
server_key_iv: "015b5051087b1d5988e1aa0b03866b1b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.723672 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "40e152f495fe46a3b5e9f353e95c3969"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "c9468cd0e4a4ac66958336b83193ff91"
server_key_iv: "015b5051087b1d5988e1aa0b03866b1b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.725332 1925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:05.726258 1940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.726512 1925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:05.726625 1925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "40e152f495fe46a3b5e9f353e95c3969"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "c9468cd0e4a4ac66958336b83193ff91"
server_key_iv: "015b5051087b1d5988e1aa0b03866b1b"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.726703 1925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:05.753242 1925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:05.755205 1925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:05.755366 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:05.755616 1925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:05.756023 1925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:05.756062 1925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.756085 1925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:05.756098 1925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.762908 1925 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:42755
I20260925 15:25:05.762952 2053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:42755 every 8 connection(s)
I20260925 15:25:05.763301 1925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:05.766199 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 1925
I20260925 15:25:05.766310 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:05.766513 1677 external_mini_cluster.cc:1428] Setting key e36ca6face8e864cbfa91c921bb9d5bb
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:05.773908 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.764080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:41275 (local address 127.1.163.126:35987)
0925 15:25:05.764145 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.764147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.764330 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:25:05.764951 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.764953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.764965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.765017 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.765982 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.766676 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.767797 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.768014 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.768101 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.770458 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.770460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.770460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.770468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.772281 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.772871 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.772872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.772873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.772938 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.773212 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.773213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.773214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.773341 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.773420 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.773722 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.773804 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:25:05.774132 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.764057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35987 (local address 127.1.163.66:41275)
0925 15:25:05.764213 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:05.764251 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:05.764834 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:05.765095 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:05.765098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:05.765682 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:05.765880 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.765885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.766765 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.766766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.767700 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.767702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.768111 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.768112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.768245 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.768880 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:05.768889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:05.770351 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:05.772422 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.772430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.772433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.772772 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.773010 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.773011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.773012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.773135 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.773411 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:05.773417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:05.773579 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:25:05.773688 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:25:05.773876 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260925 15:25:05.774690 2054 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35987
I20260925 15:25:05.774816 2054 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:05.775063 2054 heartbeater.cc:511] Master 127.1.163.126:35987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
I20260925 15:25:05.775484 1729 ts_manager.cc:194] Registered new tserver with Master: 40e152f495fe46a3b5e9f353e95c3969 (127.1.163.66:42755)
I20260925 15:25:05.775930 1729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:41275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:05.796623 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:05.883891 2059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:05.884138 2059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:05.884177 2059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:05.885388 2059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:05.885464 2059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:05.885524 2059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:05.887022 2059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:05.887305 2059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:05.887547 2059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:05.893396 2067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.893392 2064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:05.893493 2065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:05.893711 2059 server_base.cc:1034] running on GCE node
I20260925 15:25:05.893954 2059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:05.894225 2059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:05.895414 2059 hybrid_clock.cc:648] HybridClock initialized: now 1790349905895396 us; error 36 us; skew 500 ppm
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:05.898095 2059 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:05.898563 2059 webserver.cc:460] Webserver started at http://127.1.163.67:34843/ using document root <none> and password file <none>
I20260925 15:25:05.898804 2059 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:05.898874 2059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:05.899029 2059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:05.900113 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "2017eadd62604eda9d502b5df2573baa"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "804e2b3b39b58b2d6c9faddd5f38ad9c"
server_key_iv: "443299214ddad271e289c372fbe0ec04"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.900620 2059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "2017eadd62604eda9d502b5df2573baa"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "804e2b3b39b58b2d6c9faddd5f38ad9c"
server_key_iv: "443299214ddad271e289c372fbe0ec04"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.902032 2059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:05.902855 2074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.903085 2059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:05.903191 2059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "2017eadd62604eda9d502b5df2573baa"
format_stamp: "Formatted at 2026-09-25 15:25:05 on dist-test-slave-hvv5"
server_key: "804e2b3b39b58b2d6c9faddd5f38ad9c"
server_key_iv: "443299214ddad271e289c372fbe0ec04"
server_key_version: "encryptionkey@0"
I20260925 15:25:05.903277 2059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:05.916834 2059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:05.918980 2059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:05.919155 2059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:05.919409 2059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:05.919795 2059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:05.919866 2059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.919909 2059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:05.919935 2059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:05.926182 2059 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:45373
I20260925 15:25:05.926259 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:45373 every 8 connection(s)
I20260925 15:25:05.926613 2059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:05.931910 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2059
I20260925 15:25:05.932015 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:05.932174 1677 external_mini_cluster.cc:1428] Setting key aa640111139fa10746b587f7751287b6
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:05.936825 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.927339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:58841 (local address 127.1.163.126:35987)
0925 15:25:05.927384 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.927386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.927588 (+ 202us) server_negotiation.cc:407] Connection header received
0925 15:25:05.928156 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.928157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.928171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.928213 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.929288 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.929882 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.930977 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.931175 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.931264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.933938 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.933940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.933940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.933948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.935332 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.935780 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.935782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.935782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.935848 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.936130 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.936131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.936132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.936260 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.936340 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.936638 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.936732 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:25:05.937045 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.927313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35987 (local address 127.1.163.67:58841)
0925 15:25:05.927455 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:05.927499 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:05.928033 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:05.928297 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:05.928302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:05.928967 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:05.929168 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.929174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.929981 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.929982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.930895 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:05.930898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.931289 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.931290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:05.931431 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.932012 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:05.932040 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:05.933832 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:05.935446 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.935461 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.935464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.935697 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.935945 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:05.935946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:05.935946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:05.936059 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:05.936329 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:05.936334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:05.936482 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:25:05.936617 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:25:05.936787 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260925 15:25:05.937608 2188 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35987
I20260925 15:25:05.937724 2188 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:05.937924 2188 heartbeater.cc:511] Master 127.1.163.126:35987 requested a full tablet report, sending...
I20260925 15:25:05.938362 1729 ts_manager.cc:194] Registered new tserver with Master: 2017eadd62604eda9d502b5df2573baa (127.1.163.67:45373)
I20260925 15:25:05.938788 1729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:58841
I20260925 15:25:05.944828 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:05.968726 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.960756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.1.163.65:42501)
0925 15:25:05.960900 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.960905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.960997 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:25:05.961042 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.961044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.961057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.961124 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.961673 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.962422 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.963159 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.963490 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.963549 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.965934 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.965943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.965943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.965950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.967588 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.967919 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.967922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.967922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.967977 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.968156 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.968157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.968157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.968289 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.968399 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.968462 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.968525 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260925 15:25:05.969112 1834 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38198
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:25:05 dist-test-slave-hvv5 krb5kdc[1684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:05.991600 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:05.984240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.1.163.65:42501)
0925 15:25:05.984328 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:05.984330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:05.984337 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:05.984359 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:05.984360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:05.984368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:05.984436 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:05.985068 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.985562 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.986310 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:05.986515 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:05.986599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:05.989010 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:05.989011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:05.989012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:05.989018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:05.990627 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.990960 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.990961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.990961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.991018 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:05.991186 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:05.991187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:05.991187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:05.991305 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:05.991415 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:05.991464 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:25:05.991533 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:25:05.992188 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 1791
I20260925 15:25:05.998118 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 1925
I20260925 15:25:06.002790 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2059
I20260925 15:25:06.007036 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 1697
2026-09-25T15:25:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1975 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:06 dist-test-slave-hvv5 krb5kdc[2202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:06 dist-test-slave-hvv5 krb5kdc[2202](info): set up 2 sockets
Sep 25 15:25:06 dist-test-slave-hvv5 krb5kdc[2202](info): commencing operation
krb5kdc: starting...
W20260925 15:25:07.056201 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:07.184456 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38307
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:07.268016 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:07.268266 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:07.268293 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:07.269743 2215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:07.269793 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:07.269810 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:07.269824 2215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:07.269838 2215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:07.271570 2215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:07.271888 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:07.272151 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:07.277751 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.277686 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.277884 2215 server_base.cc:1034] running on GCE node
W20260925 15:25:07.277729 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.278228 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:07.278486 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:07.279654 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790349907279642 us; error 45 us; skew 500 ppm
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:07.295508 2215 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:07.296032 2215 webserver.cc:460] Webserver started at http://127.1.163.126:32875/ using document root <none> and password file <none>
I20260925 15:25:07.296291 2215 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:07.296373 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:07.296523 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:07.297701 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "2999ce40135649719db1aeb47a74d896"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "88a84fd67af413fa721d17fd6f259715"
server_key_iv: "cbfb6ca3b8ecdb20a9e50ab6e5528403"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.298079 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "2999ce40135649719db1aeb47a74d896"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "88a84fd67af413fa721d17fd6f259715"
server_key_iv: "cbfb6ca3b8ecdb20a9e50ab6e5528403"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.299656 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:07.300498 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.300858 2215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:07.300971 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "2999ce40135649719db1aeb47a74d896"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "88a84fd67af413fa721d17fd6f259715"
server_key_iv: "cbfb6ca3b8ecdb20a9e50ab6e5528403"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.301064 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:07.314479 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:07.317091 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:07.317278 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:07.321400 2215 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:38307
I20260925 15:25:07.321434 2282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:38307 every 8 connection(s)
I20260925 15:25:07.321805 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:07.322698 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:07.325287 2283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896: Bootstrap starting.
I20260925 15:25:07.325902 2283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:07.326201 2283 log.cc:824] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:07.326963 2283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896: No bootstrap required, opened a new log
I20260925 15:25:07.328679 2283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } }
I20260925 15:25:07.328846 2283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:07.328887 2283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2999ce40135649719db1aeb47a74d896, State: Initialized, Role: FOLLOWER
I20260925 15:25:07.329052 2283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } }
I20260925 15:25:07.329135 2283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:07.329180 2283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:07.329231 2283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:07.329794 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2215
I20260925 15:25:07.329802 2283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } }
I20260925 15:25:07.329865 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:07.329923 2283 leader_election.cc:302] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2999ce40135649719db1aeb47a74d896; no voters:
I20260925 15:25:07.330081 1677 external_mini_cluster.cc:1428] Setting key a28265fc50de39d058373dd7450fbd3f
I20260925 15:25:07.330109 2283 leader_election.cc:288] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:07.330171 2286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:07.330343 2286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [term 1 LEADER]: Becoming Leader. State: Replica: 2999ce40135649719db1aeb47a74d896, State: Running, Role: LEADER
I20260925 15:25:07.330391 2283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:07.330448 2286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } }
I20260925 15:25:07.331233 2286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2999ce40135649719db1aeb47a74d896. Latest consensus state: current_term: 1 leader_uuid: "2999ce40135649719db1aeb47a74d896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } } }
I20260925 15:25:07.331323 2286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:07.331329 2287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2999ce40135649719db1aeb47a74d896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2999ce40135649719db1aeb47a74d896" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38307 } } }
I20260925 15:25:07.331553 2287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:07.331583 2299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:07.332361 2299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:07.334144 2299 catalog_manager.cc:1347] Generated new cluster ID: 482ac5f38bd74cf484e292bd71198643
I20260925 15:25:07.334197 2299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:07.343096 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.331059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.1.163.126:38307)
0925 15:25:07.331722 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.331733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.331770 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:25:07.331926 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.331934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.332194 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.332355 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.333135 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.334372 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.335449 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.336076 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.336178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.339242 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.339248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.339248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.339263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.341312 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.341732 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.341735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.341735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.341864 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.342084 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.342085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.342085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.342397 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.342562 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.342675 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.342836 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":88,"threads_started":1}
I20260925 15:25:07.343869 2299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:07.344313 2299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:07.348482 2299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2999ce40135649719db1aeb47a74d896: Generated new TSK 0
I20260925 15:25:07.348713 2299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:07.377993 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:07.463685 2309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:07.463901 2309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:07.463928 2309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:07.465147 2309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:07.465193 2309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:07.465221 2309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:07.467011 2309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:07.467234 2309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:07.467430 2309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:07.473261 2315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.473441 2309 server_base.cc:1034] running on GCE node
W20260925 15:25:07.473240 2317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.473320 2314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.473938 2309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:07.474267 2309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:07.475703 2309 hybrid_clock.cc:648] HybridClock initialized: now 1790349907475568 us; error 55 us; skew 500 ppm
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:07.478945 2309 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:07.479406 2309 webserver.cc:460] Webserver started at http://127.1.163.65:45689/ using document root <none> and password file <none>
I20260925 15:25:07.479602 2309 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:07.479645 2309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:07.479740 2309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:07.480903 2309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "03537a8f8301401bb9047c886d258737"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "82359f1c5b3664df1a709d03144961c4"
server_key_iv: "266f4da76378f170d9998c551216a0d4"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.481240 2309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "03537a8f8301401bb9047c886d258737"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "82359f1c5b3664df1a709d03144961c4"
server_key_iv: "266f4da76378f170d9998c551216a0d4"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.482645 2309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:07.483423 2324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.483669 2309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:07.483765 2309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "03537a8f8301401bb9047c886d258737"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "82359f1c5b3664df1a709d03144961c4"
server_key_iv: "266f4da76378f170d9998c551216a0d4"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.483819 2309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:07.512303 2309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:07.514484 2309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:07.514605 2309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:07.514810 2309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:07.515182 2309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:07.515216 2309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.515241 2309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:07.515290 2309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.521577 2309 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:36297
I20260925 15:25:07.521723 2437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:36297 every 8 connection(s)
I20260925 15:25:07.522030 2309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:07.523595 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2309
I20260925 15:25:07.523720 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:07.523895 1677 external_mini_cluster.cc:1428] Setting key a81fb536711c4ef5305ab7293e634bee
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:07.533846 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.522895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:32995 (local address 127.1.163.126:38307)
0925 15:25:07.522959 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.522961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.523213 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:25:07.524396 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.524398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.524413 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.524510 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.525366 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.526205 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.527914 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.528199 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.528257 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.530689 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.530691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.530691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.530699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.532270 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.532787 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.532788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.532789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.532870 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.533132 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.533133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.533134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.533256 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.533331 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.533624 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.533714 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:25:07.534018 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.522858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38307 (local address 127.1.163.65:32995)
0925 15:25:07.523053 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.523098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.524192 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.524628 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.524632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.525020 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:07.525246 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.525250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.526368 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.526370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.527802 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.527805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.528290 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.528292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.528455 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.529129 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:07.529137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:07.530589 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:07.532375 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.532383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.532385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.532698 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.532943 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.532944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.532944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.533055 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.533323 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:07.533329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:07.533477 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.533591 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.533755 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 15:25:07.534570 2438 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38307
I20260925 15:25:07.534686 2438 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:07.534899 2438 heartbeater.cc:511] Master 127.1.163.126:38307 requested a full tablet report, sending...
I20260925 15:25:07.535408 2247 ts_manager.cc:194] Registered new tserver with Master: 03537a8f8301401bb9047c886d258737 (127.1.163.65:36297)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
I20260925 15:25:07.536209 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:32995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:07.552199 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:07.636852 2443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:07.637072 2443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:07.637099 2443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:07.638249 2443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:07.638288 2443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:07.638316 2443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:07.639763 2443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:07.639966 2443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:07.640149 2443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:07.645237 2451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.645283 2448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.645308 2449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.645756 2443 server_base.cc:1034] running on GCE node
I20260925 15:25:07.646008 2443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:07.646267 2443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:07.647471 2443 hybrid_clock.cc:648] HybridClock initialized: now 1790349907647467 us; error 39 us; skew 500 ppm
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:07.650182 2443 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:07.650666 2443 webserver.cc:460] Webserver started at http://127.1.163.66:41237/ using document root <none> and password file <none>
I20260925 15:25:07.650903 2443 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:07.650972 2443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:07.651139 2443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:07.652209 2443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "2be48a93dffe409d9d37e7889c1006d8"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "e8f7c44893a5dcac677598a7fee5cd35"
server_key_iv: "239814cbafa7a52d40924770bd5912cc"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.652666 2443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "2be48a93dffe409d9d37e7889c1006d8"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "e8f7c44893a5dcac677598a7fee5cd35"
server_key_iv: "239814cbafa7a52d40924770bd5912cc"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.654193 2443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:07.655136 2458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.655376 2443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:07.655483 2443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "2be48a93dffe409d9d37e7889c1006d8"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "e8f7c44893a5dcac677598a7fee5cd35"
server_key_iv: "239814cbafa7a52d40924770bd5912cc"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.655572 2443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:07.670749 2443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:07.673188 2443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:07.673363 2443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:07.673614 2443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:07.674007 2443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:07.674064 2443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.674109 2443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:07.674136 2443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.679937 2443 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:42739
I20260925 15:25:07.680023 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:42739 every 8 connection(s)
I20260925 15:25:07.680357 2443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:07.687358 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2443
I20260925 15:25:07.687460 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:07.687675 1677 external_mini_cluster.cc:1428] Setting key c2ddee62b98ff6864d5fb28dd4cfe71f
I20260925 15:25:07.691114 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.681189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:45195 (local address 127.1.163.126:38307)
0925 15:25:07.681273 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.681276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.681510 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:25:07.682519 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.682520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.682530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.682585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.683197 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.683688 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.684769 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.684973 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.685063 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.687643 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.687644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.687645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.687653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.689304 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.689850 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.689851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.689852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.689924 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.690275 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.690276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.690276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.690434 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.690525 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.690966 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.691046 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:25:07.691448 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.681173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38307 (local address 127.1.163.66:45195)
0925 15:25:07.681378 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.681418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.682424 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.682639 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.682643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.682921 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:07.683125 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.683129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.683764 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.683764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.684680 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.684683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.685098 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.685099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.685219 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.685734 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:07.685743 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:07.687519 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:07.689392 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.689402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.689405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.689765 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.690072 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.690073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.690074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.690198 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.690537 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:07.690543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:07.690732 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.690945 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.691170 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
I20260925 15:25:07.692054 2572 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38307
I20260925 15:25:07.692150 2572 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:07.692379 2572 heartbeater.cc:511] Master 127.1.163.126:38307 requested a full tablet report, sending...
I20260925 15:25:07.692886 2247 ts_manager.cc:194] Registered new tserver with Master: 2be48a93dffe409d9d37e7889c1006d8 (127.1.163.66:42739)
I20260925 15:25:07.693513 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:45195
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:07.720088 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:07.806659 2577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:07.806903 2577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:07.806931 2577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:07.808116 2577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:07.808163 2577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:07.808194 2577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:07.809804 2577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:07.810081 2577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:07.810299 2577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:25:07.816114 2577 server_base.cc:1034] running on GCE node
W20260925 15:25:07.816082 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.816118 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:07.816082 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:07.816637 2577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:07.816881 2577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:07.818038 2577 hybrid_clock.cc:648] HybridClock initialized: now 1790349907818026 us; error 38 us; skew 500 ppm
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:07.820751 2577 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:07.821297 2577 webserver.cc:460] Webserver started at http://127.1.163.67:38303/ using document root <none> and password file <none>
I20260925 15:25:07.821544 2577 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:07.821590 2577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:07.821722 2577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:07.822794 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "5cb29be1858a461bae817e7179452f3d"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "94e2f7aab8600d85a18ce84dd0f80bc2"
server_key_iv: "bbdf56ada96c8e228dcc0442163af59e"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.823167 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "5cb29be1858a461bae817e7179452f3d"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "94e2f7aab8600d85a18ce84dd0f80bc2"
server_key_iv: "bbdf56ada96c8e228dcc0442163af59e"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.824622 2577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:25:07.825640 2592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.825865 2577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:07.825992 2577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "5cb29be1858a461bae817e7179452f3d"
format_stamp: "Formatted at 2026-09-25 15:25:07 on dist-test-slave-hvv5"
server_key: "94e2f7aab8600d85a18ce84dd0f80bc2"
server_key_iv: "bbdf56ada96c8e228dcc0442163af59e"
server_key_version: "encryptionkey@0"
I20260925 15:25:07.826086 2577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:07.851428 2577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:07.853775 2577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:07.853967 2577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:07.854221 2577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:07.854588 2577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:07.854647 2577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.854684 2577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:07.854715 2577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:07.860906 2577 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:34999
I20260925 15:25:07.860947 2705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:34999 every 8 connection(s)
I20260925 15:25:07.861291 2577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:07.865953 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2577
I20260925 15:25:07.866039 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:07.866202 1677 external_mini_cluster.cc:1428] Setting key bec8dd80924a27af8ba6c267fad221e8
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:07.871055 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.862089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:49697 (local address 127.1.163.126:38307)
0925 15:25:07.862156 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.862158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.862344 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:25:07.863273 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.863275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.863285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.863343 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.863980 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.864469 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.865572 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.865758 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.865847 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.868232 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.868234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.868234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.868243 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.869652 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.870075 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.870076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.870077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.870135 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.870396 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.870397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.870398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.870518 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.870596 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.870875 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.870954 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:07.871243 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.862059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38307 (local address 127.1.163.67:49697)
0925 15:25:07.862207 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.862251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.863173 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.863408 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.863412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.863669 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:07.863870 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.863874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.864603 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.864604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.865481 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.865483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.865911 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.865911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.866041 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.866672 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:07.866686 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:07.868135 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:07.869750 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.869757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.869759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.869984 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.870207 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:07.870208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:07.870208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:07.870322 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:07.870587 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:07.870598 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:07.870737 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.870840 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.871000 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 15:25:07.871750 2706 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38307
I20260925 15:25:07.871833 2706 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:07.871970 2706 heartbeater.cc:511] Master 127.1.163.126:38307 requested a full tablet report, sending...
I20260925 15:25:07.872323 2247 ts_manager.cc:194] Registered new tserver with Master: 5cb29be1858a461bae817e7179452f3d (127.1.163.67:34999)
I20260925 15:25:07.872786 2247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:49697
I20260925 15:25:07.878153 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:07.885916 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.879831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.1.163.126:38307)
0925 15:25:07.879903 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.879906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.879916 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:07.879989 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.879990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.879999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.880061 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.880624 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.881117 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.881800 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.881996 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.882053 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.883280 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.883282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.883282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.883310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.884814 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.885181 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.885182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.885182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.885237 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.885455 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.885455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.885456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.885605 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.885680 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.885781 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.885850 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:25:07.888413 2247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:07.890183 2247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:07.897331 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.892940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.1.163.67:34999)
0925 15:25:07.893111 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.893116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.893298 (+ 182us) server_negotiation.cc:407] Connection header received
0925 15:25:07.893401 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.893403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.893419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.893543 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:07.893701 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.894614 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.896069 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.896796 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.896883 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.896945 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.897180 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.897229 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260925 15:25:07.897339 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.892872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34999 (local address 127.0.0.1:53414)
0925 15:25:07.893179 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.893219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.893317 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.893549 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.893552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.893554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:07.893629 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.893632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.894727 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.894728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.895945 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.895948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.896985 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.896986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.897080 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.897096 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.897165 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.897229 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260925 15:25:07.897341 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.893102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:36297 (local address 127.0.0.1:41964)
0925 15:25:07.893405 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.893415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.893460 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.893750 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.893751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.893753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:07.893789 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.893791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.895120 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.895122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.896165 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.896167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.897063 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.897064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.897129 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.897134 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.897182 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.897233 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260925 15:25:07.897595 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.893228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.1.163.65:36297)
0925 15:25:07.893584 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.893589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.893598 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:07.893634 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.893635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.893649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.893751 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:07.893865 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.895008 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.896244 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.896969 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.897034 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.897086 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.897441 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.897494 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":77,"threads_started":1}
I20260925 15:25:07.898164 2372 tablet_service.cc:1467] Processing CreateTablet for tablet 2a80146a58e544349f4d9d01c91379a0 (DEFAULT_TABLE table=test-table [id=845c6fc3af6f43ff9d9f89f73d854d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:07.898159 2640 tablet_service.cc:1467] Processing CreateTablet for tablet 2a80146a58e544349f4d9d01c91379a0 (DEFAULT_TABLE table=test-table [id=845c6fc3af6f43ff9d9f89f73d854d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:07.898551 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.892878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42739 (local address 127.0.0.1:58318)
0925 15:25:07.893236 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.893247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.893317 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.893648 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.893649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.893652 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:07.893790 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.893792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.896232 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.896232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.897043 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.897046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.898333 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.898334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.898399 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.898403 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.898453 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.898490 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":82,"threads_started":1}
I20260925 15:25:07.898567 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a80146a58e544349f4d9d01c91379a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:07.898567 2372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a80146a58e544349f4d9d01c91379a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:07.898584 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.892978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.1.163.66:42739)
0925 15:25:07.893186 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.893189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.893409 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:25:07.893460 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.893463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.893481 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.893552 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:07.893888 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.895008 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.897132 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.898231 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.898321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.898374 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.898463 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.898511 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260925 15:25:07.899225 2506 tablet_service.cc:1467] Processing CreateTablet for tablet 2a80146a58e544349f4d9d01c91379a0 (DEFAULT_TABLE table=test-table [id=845c6fc3af6f43ff9d9f89f73d854d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:07.899453 2506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a80146a58e544349f4d9d01c91379a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:07.901909 2723 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Bootstrap starting.
I20260925 15:25:07.902437 2725 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: Bootstrap starting.
I20260925 15:25:07.902581 2723 tablet_bootstrap.cc:654] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:07.903106 2725 tablet_bootstrap.cc:654] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:07.903281 2723 log.cc:824] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:07.903648 2725 log.cc:824] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:07.903954 2724 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: Bootstrap starting.
I20260925 15:25:07.904088 2723 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: No bootstrap required, opened a new log
I20260925 15:25:07.904177 2723 ts_tablet_manager.cc:1397] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:07.904650 2724 tablet_bootstrap.cc:654] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:07.904983 2725 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: No bootstrap required, opened a new log
I20260925 15:25:07.905059 2725 ts_tablet_manager.cc:1397] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:25:07.905186 2724 log.cc:824] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:07.906023 2724 tablet_bootstrap.cc:492] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: No bootstrap required, opened a new log
I20260925 15:25:07.906095 2724 ts_tablet_manager.cc:1397] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:25:07.906220 2723 raft_consensus.cc:348] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.906391 2723 raft_consensus.cc:374] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:07.906433 2723 raft_consensus.cc:729] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be48a93dffe409d9d37e7889c1006d8, State: Initialized, Role: FOLLOWER
I20260925 15:25:07.906574 2723 consensus_queue.cc:260] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.906870 2572 heartbeater.cc:503] Master 127.1.163.126:38307 was elected leader, sending a full tablet report...
I20260925 15:25:07.906883 2723 ts_tablet_manager.cc:1428] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:07.907028 2725 raft_consensus.cc:348] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.907195 2725 raft_consensus.cc:374] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:07.907238 2725 raft_consensus.cc:729] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb29be1858a461bae817e7179452f3d, State: Initialized, Role: FOLLOWER
I20260925 15:25:07.907346 2725 consensus_queue.cc:260] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.907630 2706 heartbeater.cc:503] Master 127.1.163.126:38307 was elected leader, sending a full tablet report...
I20260925 15:25:07.907621 2725 ts_tablet_manager.cc:1428] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:07.908371 2724 raft_consensus.cc:348] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.908537 2724 raft_consensus.cc:374] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:07.908615 2724 raft_consensus.cc:729] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03537a8f8301401bb9047c886d258737, State: Initialized, Role: FOLLOWER
I20260925 15:25:07.908725 2724 consensus_queue.cc:260] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.909008 2724 ts_tablet_manager.cc:1428] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:25:07.909034 2438 heartbeater.cc:503] Master 127.1.163.126:38307 was elected leader, sending a full tablet report...
W20260925 15:25:07.930986 2573 tablet.cc:2369] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:07.934255 2730 raft_consensus.cc:482] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:07.934369 2730 raft_consensus.cc:504] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.934805 2730 leader_election.cc:288] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03537a8f8301401bb9047c886d258737 (127.1.163.65:36297), 2be48a93dffe409d9d37e7889c1006d8 (127.1.163.66:42739)
I20260925 15:25:07.938748 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.935038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:50895 (local address 127.1.163.66:42739)
0925 15:25:07.935084 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.935085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.935097 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:07.935127 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.935128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.935141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.935202 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:07.935419 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.936194 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.937783 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.938370 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.938443 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.938452 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.938632 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.938673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:07.938748 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.935077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:43549 (local address 127.1.163.65:36297)
0925 15:25:07.935118 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.935119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.935258 (+ 139us) server_negotiation.cc:407] Connection header received
0925 15:25:07.935288 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.935289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.935298 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.935349 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:07.935494 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.936194 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.937784 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.938361 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.938443 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.938452 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.938637 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.938673 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:25:07.938786 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.934958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42739 (local address 127.1.163.67:50895)
0925 15:25:07.935001 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.935015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.935061 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.935234 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.935235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.935237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:07.935310 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.935312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.936289 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.936290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.937673 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.937676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.938454 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.938455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.938548 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.938553 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.938613 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.938665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":136}
I20260925 15:25:07.938794 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.934958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:36297 (local address 127.1.163.67:43549)
0925 15:25:07.935162 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:07.935175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:07.935244 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:07.935367 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:07.935368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:07.935370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:07.935429 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.935435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.936289 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.936290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.937673 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:07.937676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.938454 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.938455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:07.938548 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.938554 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:25:07.938613 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:25:07.938665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":103,"thread_start_us":116,"threads_started":1}
I20260925 15:25:07.939194 2526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a80146a58e544349f4d9d01c91379a0" candidate_uuid: "5cb29be1858a461bae817e7179452f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be48a93dffe409d9d37e7889c1006d8" is_pre_election: true
I20260925 15:25:07.939396 2526 raft_consensus.cc:2454] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb29be1858a461bae817e7179452f3d in term 0.
I20260925 15:25:07.939327 2392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a80146a58e544349f4d9d01c91379a0" candidate_uuid: "5cb29be1858a461bae817e7179452f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03537a8f8301401bb9047c886d258737" is_pre_election: true
I20260925 15:25:07.939527 2392 raft_consensus.cc:2454] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb29be1858a461bae817e7179452f3d in term 0.
I20260925 15:25:07.939670 2594 leader_election.cc:302] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be48a93dffe409d9d37e7889c1006d8, 5cb29be1858a461bae817e7179452f3d; no voters:
I20260925 15:25:07.939822 2730 raft_consensus.cc:2781] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:07.939884 2730 raft_consensus.cc:482] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:07.939921 2730 raft_consensus.cc:3037] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:07.940615 2730 raft_consensus.cc:504] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.940757 2730 leader_election.cc:288] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [CANDIDATE]: Term 1 election: Requested vote from peers 03537a8f8301401bb9047c886d258737 (127.1.163.65:36297), 2be48a93dffe409d9d37e7889c1006d8 (127.1.163.66:42739)
I20260925 15:25:07.940919 2392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a80146a58e544349f4d9d01c91379a0" candidate_uuid: "5cb29be1858a461bae817e7179452f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03537a8f8301401bb9047c886d258737"
I20260925 15:25:07.940932 2526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a80146a58e544349f4d9d01c91379a0" candidate_uuid: "5cb29be1858a461bae817e7179452f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be48a93dffe409d9d37e7889c1006d8"
I20260925 15:25:07.941015 2392 raft_consensus.cc:3037] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:07.941033 2526 raft_consensus.cc:3037] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:07.941745 2392 raft_consensus.cc:2454] T 2a80146a58e544349f4d9d01c91379a0 P 03537a8f8301401bb9047c886d258737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb29be1858a461bae817e7179452f3d in term 1.
I20260925 15:25:07.941759 2526 raft_consensus.cc:2454] T 2a80146a58e544349f4d9d01c91379a0 P 2be48a93dffe409d9d37e7889c1006d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb29be1858a461bae817e7179452f3d in term 1.
I20260925 15:25:07.941929 2595 leader_election.cc:302] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03537a8f8301401bb9047c886d258737, 5cb29be1858a461bae817e7179452f3d; no voters:
I20260925 15:25:07.942041 2730 raft_consensus.cc:2781] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:07.942196 2730 raft_consensus.cc:686] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [term 1 LEADER]: Becoming Leader. State: Replica: 5cb29be1858a461bae817e7179452f3d, State: Running, Role: LEADER
I20260925 15:25:07.942308 2730 consensus_queue.cc:237] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } }
I20260925 15:25:07.943230 2247 catalog_manager.cc:5495] T 2a80146a58e544349f4d9d01c91379a0 P 5cb29be1858a461bae817e7179452f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb29be1858a461bae817e7179452f3d (127.1.163.67). New cstate: current_term: 1 leader_uuid: "5cb29be1858a461bae817e7179452f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb29be1858a461bae817e7179452f3d" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03537a8f8301401bb9047c886d258737" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be48a93dffe409d9d37e7889c1006d8" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42739 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:07.964811 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.957290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.1.163.65:36297)
0925 15:25:07.957362 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.957364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.957481 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:25:07.957557 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.957558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.957570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.957639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.958287 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.958778 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.959548 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.959743 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.959849 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.962280 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.962291 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.962291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.962298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.963783 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.964099 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.964102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.964102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.964156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.964315 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.964316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.964316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.964447 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.964598 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.964672 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.964740 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:07.988945 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:07.981480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.1.163.65:36297)
0925 15:25:07.981560 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:07.981562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:07.981575 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:25:07.981680 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:07.981681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:07.981691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:07.981747 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:07.982335 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.982808 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.983561 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:07.983745 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:07.983801 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:07.986234 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:07.986235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:07.986235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:07.986243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:07.987729 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.988115 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.988116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.988116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.988169 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:07.988348 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:07.988349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:07.988349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:07.988441 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:07.988514 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:07.988772 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:25:07.988877 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:25:07 dist-test-slave-hvv5 krb5kdc[2202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:25:08 dist-test-slave-hvv5 krb5kdc[2202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:08.012214 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:08.004955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.1.163.65:36297)
0925 15:25:08.005055 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:08.005057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:08.005068 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:08.005147 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:08.005147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:08.005157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:08.005211 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:08.005803 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:08.006269 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:08.007072 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:08.007256 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:08.007312 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:08.009653 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:08.009654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:08.009655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:08.009662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:08.011141 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:08.011463 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:08.011463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:08.011464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:08.011518 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:08.011687 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:08.011687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:08.011688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:08.011783 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:08.011855 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:08.012076 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:25:08.012141 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 15:25:08.012786 2352 tablet_service.cc:2908] Rejecting scan request for tablet 2a80146a58e544349f4d9d01c91379a0: Uninitialized: clean time has not yet been initialized
I20260925 15:25:08.013769 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2309
I20260925 15:25:08.019793 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2443
I20260925 15:25:08.025265 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2577
I20260925 15:25:08.030331 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2215
2026-09-25T15:25:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2023 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:08 dist-test-slave-hvv5 krb5kdc[2747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:08 dist-test-slave-hvv5 krb5kdc[2747](info): set up 2 sockets
Sep 25 15:25:08 dist-test-slave-hvv5 krb5kdc[2747](info): commencing operation
krb5kdc: starting...
W20260925 15:25:09.092247 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:09.216248 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:40445
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:09.299914 2760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:09.300153 2760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:09.300177 2760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:09.301376 2760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:09.301416 2760 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:25:09.301429 2760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:09.301442 2760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:09.301455 2760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:09.301466 2760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:09.302948 2760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:40445
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:09.303158 2760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:09.303347 2760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:09.308427 2765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.308437 2768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.308436 2766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.308494 2760 server_base.cc:1034] running on GCE node
I20260925 15:25:09.309012 2760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:09.309346 2760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:09.310546 2760 hybrid_clock.cc:648] HybridClock initialized: now 1790349909310512 us; error 59 us; skew 500 ppm
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:09.313211 2760 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:09.313686 2760 webserver.cc:460] Webserver started at http://127.1.163.126:40449/ using document root <none> and password file <none>
I20260925 15:25:09.313907 2760 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:09.313951 2760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:09.314039 2760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:09.315454 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "4ae59610434d4f8c8d365d99d30a2719"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "473b52f9b49064791c88b4cfe8b6a339"
server_key_iv: "e71c8b2d7b43f862097f39f1a01c48c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.315816 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "4ae59610434d4f8c8d365d99d30a2719"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "473b52f9b49064791c88b4cfe8b6a339"
server_key_iv: "e71c8b2d7b43f862097f39f1a01c48c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.317318 2760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:09.318173 2775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.318446 2760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:09.318557 2760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "4ae59610434d4f8c8d365d99d30a2719"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "473b52f9b49064791c88b4cfe8b6a339"
server_key_iv: "e71c8b2d7b43f862097f39f1a01c48c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.318636 2760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:09.344965 2760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:09.348079 2760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:09.348232 2760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:09.354131 2760 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:40445
I20260925 15:25:09.354184 2837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:40445 every 8 connection(s)
I20260925 15:25:09.354599 2760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:09.355569 2838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:09.358052 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Bootstrap starting.
I20260925 15:25:09.359001 2838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:09.359334 2838 log.cc:824] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:09.360189 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: No bootstrap required, opened a new log
I20260925 15:25:09.362035 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2760
I20260925 15:25:09.362164 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:09.362334 1677 external_mini_cluster.cc:1428] Setting key 6d1178d39eba4e5336a29ee5c29c8913
I20260925 15:25:09.363224 2838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } }
I20260925 15:25:09.363741 2838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:09.364012 2838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae59610434d4f8c8d365d99d30a2719, State: Initialized, Role: FOLLOWER
I20260925 15:25:09.364187 2838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } }
I20260925 15:25:09.364303 2838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:09.364346 2838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:09.364738 2838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:09.365490 2838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } }
I20260925 15:25:09.365628 2838 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae59610434d4f8c8d365d99d30a2719; no voters:
I20260925 15:25:09.365834 2838 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:09.365869 2843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:09.365991 2843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae59610434d4f8c8d365d99d30a2719, State: Running, Role: LEADER
I20260925 15:25:09.366137 2843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } }
I20260925 15:25:09.366145 2838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:09.366624 2844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae59610434d4f8c8d365d99d30a2719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } } }
I20260925 15:25:09.366720 2844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:09.366640 2845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae59610434d4f8c8d365d99d30a2719. Latest consensus state: current_term: 1 leader_uuid: "4ae59610434d4f8c8d365d99d30a2719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae59610434d4f8c8d365d99d30a2719" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40445 } } }
I20260925 15:25:09.366921 2845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:09.366921 2847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:09.367517 2847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:09.369760 2847 catalog_manager.cc:1347] Generated new cluster ID: 7946e9908d9a4ceeb20daa7db2fce516
I20260925 15:25:09.370535 2847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:09.374409 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.362973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.1.163.126:40445)
0925 15:25:09.363248 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.363257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.363287 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:25:09.363397 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.363404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.363471 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.363647 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.364479 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.365571 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.366680 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.367329 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.367437 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.370515 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.370519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.370520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.370532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.372592 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.373001 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.373004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.373004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.373135 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.373336 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.373337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.373338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.373735 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.373940 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.374024 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.374173 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260925 15:25:09.395495 2847 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:09.396268 2847 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:09.404925 2847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Generated new TSK 0
I20260925 15:25:09.405148 2847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:09.440886 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40445
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:09.525063 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:09.525316 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:09.525357 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:09.526616 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:09.526693 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:09.526733 2864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:09.526793 2864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:09.528333 2864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:09.528640 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:09.528888 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:09.534780 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.534786 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.534806 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.534996 2864 server_base.cc:1034] running on GCE node
I20260925 15:25:09.535256 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:09.535548 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:09.536798 2864 hybrid_clock.cc:648] HybridClock initialized: now 1790349909536772 us; error 41 us; skew 500 ppm
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:09.539530 2864 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:09.540067 2864 webserver.cc:460] Webserver started at http://127.1.163.65:44975/ using document root <none> and password file <none>
I20260925 15:25:09.540310 2864 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:09.540383 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:09.540585 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:09.541782 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "baa85b82d3f54f74b437948554d05024"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "b0ca3106570e671e710e78ff2676124f"
server_key_iv: "5a673992453c70a652b166582fa0207b"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.542166 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "baa85b82d3f54f74b437948554d05024"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "b0ca3106570e671e710e78ff2676124f"
server_key_iv: "5a673992453c70a652b166582fa0207b"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.543622 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:09.544497 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.544713 2864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:09.544862 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "baa85b82d3f54f74b437948554d05024"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "b0ca3106570e671e710e78ff2676124f"
server_key_iv: "5a673992453c70a652b166582fa0207b"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.544953 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:09.566330 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:09.569024 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:09.569207 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:09.569862 2864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:09.569917 2864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.569957 2864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:09.569976 2864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:09.577615 2864 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:36521
I20260925 15:25:09.577800 2994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:36521 every 8 connection(s)
I20260925 15:25:09.578063 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:09.581537 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.570369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:60323 (local address 127.1.163.126:40445)
0925 15:25:09.570450 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.570453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.570705 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:25:09.571286 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.571288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.571316 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.571398 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.572423 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.573371 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.574633 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.575003 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.575097 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.578128 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.578131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.578131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.578147 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.579798 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.580299 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.580301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.580301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.580372 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.580860 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.580861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.580861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.581008 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.581068 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.581386 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.581470 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:25:09.581677 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.570360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.65:60323)
0925 15:25:09.570560 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.570603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.571147 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.571513 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.571529 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.571957 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.572279 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.572287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.573486 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.573488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.574549 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.574552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.575121 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.575122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.575281 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.575885 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.575893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.577985 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.579933 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.579941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.579943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.580175 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.580653 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.580654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.580654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.580779 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.581079 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.581084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.581232 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.581325 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.581473 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260925 15:25:09.582250 2995 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40445
I20260925 15:25:09.582366 2995 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:09.582551 2995 heartbeater.cc:511] Master 127.1.163.126:40445 requested a full tablet report, sending...
I20260925 15:25:09.583153 2791 ts_manager.cc:194] Registered new tserver with Master: baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521)
I20260925 15:25:09.584108 2791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:60323
I20260925 15:25:09.586668 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 2864
I20260925 15:25:09.586796 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:09.586979 1677 external_mini_cluster.cc:1428] Setting key 9ae01b2c7d244d345b2452d50c5c3865
I20260925 15:25:09.589787 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.583416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:40521 (local address 127.1.163.126:40445)
0925 15:25:09.583463 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.583465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.583607 (+ 142us) server_negotiation.cc:407] Connection header received
0925 15:25:09.583676 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.583677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.583685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.583729 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.584075 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.584742 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.585515 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.585698 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.585763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.586778 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.586780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.586780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.586788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.588380 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.588881 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.588883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.588883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.588954 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.589205 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.589206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.589206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.589334 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.589423 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.589602 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.589688 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:09.589814 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.583388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.65:40521)
0925 15:25:09.583547 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.583560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.583622 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.583734 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.583735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.583932 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.584009 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.584011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.584829 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.584830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.585429 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.585432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.585774 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.585775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.585835 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.586323 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.586328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.586649 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.588496 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.588497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.588498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.588719 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.589045 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.589046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.589047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.589118 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.589412 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.589413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.589466 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.589572 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.589698 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:09.616765 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40445
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:09.706207 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:09.706439 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:09.706467 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:09.707684 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:09.707726 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:09.707747 3004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:09.707765 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:09.709214 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:09.709407 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:09.709591 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:09.715052 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.715059 3004 server_base.cc:1034] running on GCE node
W20260925 15:25:09.715032 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.715032 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.715597 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:09.715858 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:09.717037 3004 hybrid_clock.cc:648] HybridClock initialized: now 1790349909717000 us; error 59 us; skew 500 ppm
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:09.719929 3004 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:09.720498 3004 webserver.cc:460] Webserver started at http://127.1.163.66:33235/ using document root <none> and password file <none>
I20260925 15:25:09.720858 3004 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:09.720932 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:09.721074 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:09.722251 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "3461635441b04027b6ec30189df06d21"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "24503cdf4c22cd6e0b422add0c5bd07f"
server_key_iv: "3b9c3e2532af46da77bd57cbe8e251ef"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.722636 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "3461635441b04027b6ec30189df06d21"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "24503cdf4c22cd6e0b422add0c5bd07f"
server_key_iv: "3b9c3e2532af46da77bd57cbe8e251ef"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.724076 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:09.724961 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.725159 3004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:09.725256 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "3461635441b04027b6ec30189df06d21"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "24503cdf4c22cd6e0b422add0c5bd07f"
server_key_iv: "3b9c3e2532af46da77bd57cbe8e251ef"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.725343 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:09.750167 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:09.752725 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:09.752902 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:09.753566 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:09.753618 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.753654 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:09.753676 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:09.761027 3004 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:42457
I20260925 15:25:09.761116 3134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:42457 every 8 connection(s)
I20260925 15:25:09.761425 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:09.762264 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3004
I20260925 15:25:09.762365 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:09.762543 1677 external_mini_cluster.cc:1428] Setting key 0e7a16f56608e744216800f72671fa55
I20260925 15:25:09.764343 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.754034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:37831 (local address 127.1.163.126:40445)
0925 15:25:09.754110 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.754114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.754426 (+ 312us) server_negotiation.cc:407] Connection header received
0925 15:25:09.755022 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.755024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.755038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.755117 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.756028 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.756821 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.757870 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.758089 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.758185 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.760934 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.760937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.760937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.760946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.762796 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.763255 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.763256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.763257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.763325 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.763621 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.763621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.763622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.763733 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.763784 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.764176 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.764255 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:25:09.764596 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.754008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.66:37831)
0925 15:25:09.754267 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.754321 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.754831 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.755199 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.755204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.755604 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.755930 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.755936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.756929 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.756930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.757769 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.757771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.758210 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.758211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.758346 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.758968 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.758977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.760819 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.762874 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.762880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.762883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.763181 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.763395 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.763396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.763396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.763549 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.763799 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.763805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.763990 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.764109 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.764295 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":157,"threads_started":1}
I20260925 15:25:09.765130 3135 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40445
I20260925 15:25:09.765220 3135 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:09.765457 3135 heartbeater.cc:511] Master 127.1.163.126:40445 requested a full tablet report, sending...
I20260925 15:25:09.765910 2791 ts_manager.cc:194] Registered new tserver with Master: 3461635441b04027b6ec30189df06d21 (127.1.163.66:42457)
I20260925 15:25:09.766496 2791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:37831
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
I20260925 15:25:09.773901 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.766300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:38745 (local address 127.1.163.126:40445)
0925 15:25:09.766338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.766340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.766444 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:25:09.766490 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.766491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.766501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.766580 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.767837 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.768378 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.769594 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.769810 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.769876 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.771038 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.771040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.771040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.771047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.772588 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.772938 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.772939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.772940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.773023 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.773242 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.773243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.773244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.773418 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.773550 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.773715 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.773798 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:25:09.773913 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.766233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.66:38745)
0925 15:25:09.766376 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.766387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.766442 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.766633 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.766634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.767594 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.767674 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.767677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.768488 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.768489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.769492 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.769495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.769897 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.769898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.769992 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.770492 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.770499 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.770928 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.772680 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.772681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.772682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.772848 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.773094 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.773095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.773095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.773181 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.773544 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.773545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.773595 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.773688 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.773799 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:09.792155 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40445
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:09.875298 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:09.875547 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:09.875589 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:09.876863 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:09.876927 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:09.876964 3144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:09.877022 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:09.878510 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:09.878777 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:09.879032 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:09.884636 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:09.884631 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.884672 3144 server_base.cc:1034] running on GCE node
W20260925 15:25:09.884636 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:09.885130 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:09.885406 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:09.886579 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790349909886577 us; error 41 us; skew 500 ppm
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:09.889122 3144 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:09.889560 3144 webserver.cc:460] Webserver started at http://127.1.163.67:43941/ using document root <none> and password file <none>
I20260925 15:25:09.889791 3144 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:09.889863 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:09.890018 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:09.891219 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "452c5e5bb2834c47b1f19845faa4f225"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "9aa64be5f976b27ea46a6b68c1a626e1"
server_key_iv: "c249b300f15a86107fd4ed0d30b52325"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.891631 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "452c5e5bb2834c47b1f19845faa4f225"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "9aa64be5f976b27ea46a6b68c1a626e1"
server_key_iv: "c249b300f15a86107fd4ed0d30b52325"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.893272 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:09.894088 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.894284 3144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:09.894418 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "452c5e5bb2834c47b1f19845faa4f225"
format_stamp: "Formatted at 2026-09-25 15:25:09 on dist-test-slave-hvv5"
server_key: "9aa64be5f976b27ea46a6b68c1a626e1"
server_key_iv: "c249b300f15a86107fd4ed0d30b52325"
server_key_version: "encryptionkey@0"
I20260925 15:25:09.894496 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:09.918601 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:09.920980 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:09.921164 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:09.921756 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:09.921799 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:09.921830 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:09.921847 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:09.929790 3144 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:39985
I20260925 15:25:09.929868 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:39985 every 8 connection(s)
I20260925 15:25:09.930245 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:09.932776 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.922201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:59805 (local address 127.1.163.126:40445)
0925 15:25:09.922274 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.922276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.922502 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:25:09.922993 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.922994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.923008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.923099 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.924049 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.924927 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.926081 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.926308 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.926405 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.929443 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.929445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.929445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.929454 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.931175 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.931700 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.931702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.931702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.931775 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.932045 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.932046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.932046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.932181 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.932269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.932596 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.932683 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:25:09.932962 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.922188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.67:59805)
0925 15:25:09.922366 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.922406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.922891 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.923178 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.923181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.923603 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.923922 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.923931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.925036 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.925037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.925953 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.925957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.926468 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.926469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.926598 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.927209 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.927224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.929311 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.931296 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.931304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.931306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.931612 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.931848 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.931849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.931849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.931972 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.932259 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.932266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.932417 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.932531 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.932724 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260925 15:25:09.933630 3275 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40445
I20260925 15:25:09.933724 3275 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:09.933985 3275 heartbeater.cc:511] Master 127.1.163.126:40445 requested a full tablet report, sending...
I20260925 15:25:09.934382 2791 ts_manager.cc:194] Registered new tserver with Master: 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985)
I20260925 15:25:09.934911 2791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:59805
I20260925 15:25:09.937745 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3144
I20260925 15:25:09.937857 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:09.938009 1677 external_mini_cluster.cc:1428] Setting key b08c61cfd35c98548e404142eb8c0ccb
I20260925 15:25:09.938848 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:09.940825 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.934697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.1.163.67:39599)
0925 15:25:09.934828 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.934840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.934908 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.935072 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.935073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.935289 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:09.935366 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.935368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.936139 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.936140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.936738 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.936740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.937084 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.937085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.937166 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.937571 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:09.937576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:09.937984 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:09.939630 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.939631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.939632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.939880 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.940119 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:09.940120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:09.940120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:09.940192 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:09.940406 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:09.940406 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:09.940459 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.940559 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.940700 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260925 15:25:09.940864 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.934781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:39599 (local address 127.1.163.126:40445)
0925 15:25:09.934824 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.934826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.934897 (+ 71us) server_negotiation.cc:407] Connection header received
0925 15:25:09.934970 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.934970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.934979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.935052 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.935474 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.936047 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.936806 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.936997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.937057 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.938073 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.938074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.938075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.938082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.939544 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.940001 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.940002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.940002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.940055 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.940253 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.940254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.940254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.940349 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.940398 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.940643 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.940741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:25:09 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:09.963292 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.955581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.1.163.126:40445)
0925 15:25:09.955652 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.955654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.955664 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:09.955745 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.955746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.955756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.955809 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:09.956603 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.957181 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.958005 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.958226 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.958299 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.960817 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:09.960819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:09.960819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:09.960826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:09.962120 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.962486 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.962488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.962489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.962548 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:09.962757 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:09.962758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:09.962758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:09.962873 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:09.962980 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.963097 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.963181 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:25:09.965328 2791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:09.967232 2791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:09.974118 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.969772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42457 (local address 127.0.0.1:35236)
0925 15:25:09.969905 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.969943 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.970031 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.970710 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.970711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.970713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:09.970750 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.970751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.971632 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.971633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.972844 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.972847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.973805 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.973806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.973889 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.973904 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.973977 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.974034 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260925 15:25:09.974164 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.970065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.1.163.66:42457)
0925 15:25:09.970566 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.970570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.970579 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:09.970617 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.970619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.970632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.970700 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:09.970830 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.971540 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.972961 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.973711 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.973793 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.973858 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.974006 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.974054 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":92,"threads_started":1}
I20260925 15:25:09.974488 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.969775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:39985 (local address 127.0.0.1:42238)
0925 15:25:09.970243 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.970253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.970435 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.970702 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.970703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.970705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:09.970761 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.970763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.972049 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.972050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.973101 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.973103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.974119 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.974120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.974195 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.974310 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.974359 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.974410 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":113,"threads_started":1}
I20260925 15:25:09.974532 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.970061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.1.163.67:39985)
0925 15:25:09.970229 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.970234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.970349 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:25:09.970517 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.970521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.970538 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.970615 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:09.970844 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.971957 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.973182 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.973939 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.974026 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.974081 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.974396 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.974438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 15:25:09.975030 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.969772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:36521 (local address 127.0.0.1:34370)
0925 15:25:09.970327 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:09.970337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:09.970392 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:09.970654 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:09.970657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:09.970660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:09.970726 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.970728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.973174 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.973174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.973904 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:09.973905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.974788 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.974789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:09.974861 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.974866 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:09.974923 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:25:09.974967 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260925 15:25:09.975085 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:09.970065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.1.163.65:36521)
0925 15:25:09.970309 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:09.970313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:09.970379 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:25:09.970463 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:09.970466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:09.970483 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:09.970563 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:09.970799 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.971955 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.974003 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:09.974716 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:09.974788 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:09.974840 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:09.974947 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:25:09.974984 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260925 15:25:09.975281 3209 tablet_service.cc:1467] Processing CreateTablet for tablet cf6529f9c9d84be09eb8576fce83a577 (DEFAULT_TABLE table=test-table [id=399349988bf2433b934921cffcebdafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:09.975672 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6529f9c9d84be09eb8576fce83a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:09.975777 2929 tablet_service.cc:1467] Processing CreateTablet for tablet cf6529f9c9d84be09eb8576fce83a577 (DEFAULT_TABLE table=test-table [id=399349988bf2433b934921cffcebdafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:09.976076 2929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6529f9c9d84be09eb8576fce83a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:09.978694 3298 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Bootstrap starting.
I20260925 15:25:09.979140 3299 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Bootstrap starting.
I20260925 15:25:09.979597 3069 tablet_service.cc:1467] Processing CreateTablet for tablet cf6529f9c9d84be09eb8576fce83a577 (DEFAULT_TABLE table=test-table [id=399349988bf2433b934921cffcebdafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:09.979769 3298 tablet_bootstrap.cc:654] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:09.979769 3299 tablet_bootstrap.cc:654] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:09.979938 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6529f9c9d84be09eb8576fce83a577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:09.980129 3298 log.cc:824] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:09.980129 3299 log.cc:824] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:09.981143 3299 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: No bootstrap required, opened a new log
I20260925 15:25:09.981222 3299 ts_tablet_manager.cc:1397] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:09.981143 3298 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: No bootstrap required, opened a new log
I20260925 15:25:09.981554 3298 ts_tablet_manager.cc:1397] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:25:09.983191 3302 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Bootstrap starting.
I20260925 15:25:09.983458 3299 raft_consensus.cc:348] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.983645 3299 raft_consensus.cc:374] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:09.983681 3299 raft_consensus.cc:729] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa85b82d3f54f74b437948554d05024, State: Initialized, Role: FOLLOWER
I20260925 15:25:09.983608 3298 raft_consensus.cc:348] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.983741 3298 raft_consensus.cc:374] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:09.983772 3298 raft_consensus.cc:729] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452c5e5bb2834c47b1f19845faa4f225, State: Initialized, Role: FOLLOWER
I20260925 15:25:09.983803 3299 consensus_queue.cc:260] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.983861 3302 tablet_bootstrap.cc:654] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:09.983883 3298 consensus_queue.cc:260] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.984087 3299 ts_tablet_manager.cc:1428] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:09.984148 3298 ts_tablet_manager.cc:1428] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:25:09.984211 3302 log.cc:824] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:09.984230 2995 heartbeater.cc:503] Master 127.1.163.126:40445 was elected leader, sending a full tablet report...
I20260925 15:25:09.984249 3275 heartbeater.cc:503] Master 127.1.163.126:40445 was elected leader, sending a full tablet report...
I20260925 15:25:09.985031 3302 tablet_bootstrap.cc:492] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: No bootstrap required, opened a new log
I20260925 15:25:09.985114 3302 ts_tablet_manager.cc:1397] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:09.986683 3302 raft_consensus.cc:348] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.986846 3302 raft_consensus.cc:374] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:09.986893 3302 raft_consensus.cc:729] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3461635441b04027b6ec30189df06d21, State: Initialized, Role: FOLLOWER
I20260925 15:25:09.987008 3302 consensus_queue.cc:260] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:09.987277 3302 ts_tablet_manager.cc:1428] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:25:09.987362 3135 heartbeater.cc:503] Master 127.1.163.126:40445 was elected leader, sending a full tablet report...
W20260925 15:25:10.012120 3136 tablet.cc:2369] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:10.033548 3304 raft_consensus.cc:482] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:10.033691 3304 raft_consensus.cc:504] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:10.034106 3304 leader_election.cc:288] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 3461635441b04027b6ec30189df06d21 (127.1.163.66:42457)
I20260925 15:25:10.037104 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.034417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53845 (local address 127.1.163.65:36521)
0925 15:25:10.034512 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.034514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.034526 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:10.034556 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.034557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.034568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.034638 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:10.034828 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.035340 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.036323 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.036741 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.036817 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.036826 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.037018 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.037051 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:25:10.037118 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.034327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:36521 (local address 127.1.163.67:53845)
0925 15:25:10.034392 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.034406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.034468 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.034686 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.034687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.034689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:10.034747 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.034748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.035431 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.035432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.036221 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.036224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.036827 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.036828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.036917 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.036922 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.036994 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.037037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:25:10.037561 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.034408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42457 (local address 127.1.163.67:44389)
0925 15:25:10.034621 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.034630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.034687 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.034864 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.034865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.034866 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:10.034945 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.034952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.035636 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.035637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.036400 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.036402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.037283 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.037284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.037368 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.037373 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.037428 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.037470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":155,"threads_started":1}
I20260925 15:25:10.037554 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.034508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:44389 (local address 127.1.163.66:42457)
0925 15:25:10.034555 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.034556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.034675 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:25:10.034756 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.034757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.034768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.034847 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:10.035027 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.035529 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.036514 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.037133 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.037210 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.037220 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.037447 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.037491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:10.037519 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6529f9c9d84be09eb8576fce83a577" candidate_uuid: "452c5e5bb2834c47b1f19845faa4f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024" is_pre_election: true
I20260925 15:25:10.037693 2949 raft_consensus.cc:2454] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 452c5e5bb2834c47b1f19845faa4f225 in term 0.
I20260925 15:25:10.037956 3163 leader_election.cc:302] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 452c5e5bb2834c47b1f19845faa4f225, baa85b82d3f54f74b437948554d05024; no voters:
I20260925 15:25:10.037935 3089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6529f9c9d84be09eb8576fce83a577" candidate_uuid: "452c5e5bb2834c47b1f19845faa4f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3461635441b04027b6ec30189df06d21" is_pre_election: true
I20260925 15:25:10.038106 3304 raft_consensus.cc:2781] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:10.038113 3089 raft_consensus.cc:2454] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 452c5e5bb2834c47b1f19845faa4f225 in term 0.
I20260925 15:25:10.038172 3304 raft_consensus.cc:482] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:10.038208 3304 raft_consensus.cc:3037] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.038949 3304 raft_consensus.cc:504] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:10.039114 3304 leader_election.cc:288] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [CANDIDATE]: Term 1 election: Requested vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 3461635441b04027b6ec30189df06d21 (127.1.163.66:42457)
I20260925 15:25:10.039343 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6529f9c9d84be09eb8576fce83a577" candidate_uuid: "452c5e5bb2834c47b1f19845faa4f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024"
I20260925 15:25:10.039343 3089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6529f9c9d84be09eb8576fce83a577" candidate_uuid: "452c5e5bb2834c47b1f19845faa4f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3461635441b04027b6ec30189df06d21"
I20260925 15:25:10.039443 2949 raft_consensus.cc:3037] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.039443 3089 raft_consensus.cc:3037] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.040441 3089 raft_consensus.cc:2454] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 452c5e5bb2834c47b1f19845faa4f225 in term 1.
I20260925 15:25:10.040441 2949 raft_consensus.cc:2454] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 452c5e5bb2834c47b1f19845faa4f225 in term 1.
I20260925 15:25:10.040972 3161 leader_election.cc:302] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3461635441b04027b6ec30189df06d21, 452c5e5bb2834c47b1f19845faa4f225; no voters:
I20260925 15:25:10.041116 3304 raft_consensus.cc:2781] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:10.041317 3304 raft_consensus.cc:686] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 LEADER]: Becoming Leader. State: Replica: 452c5e5bb2834c47b1f19845faa4f225, State: Running, Role: LEADER
I20260925 15:25:10.041426 3304 consensus_queue.cc:237] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } }
I20260925 15:25:10.042354 2792 catalog_manager.cc:5495] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "452c5e5bb2834c47b1f19845faa4f225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:10.048822 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.046108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.1.163.67:39985)
0925 15:25:10.046164 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.046166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.046175 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:10.046232 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.046233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.046251 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.046310 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:10.046500 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.046982 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.047815 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.048125 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.048200 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.048423 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.048569 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.048642 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.048696 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.048721 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:25:10.050711 3089 raft_consensus.cc:1261] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Refusing update from remote peer 452c5e5bb2834c47b1f19845faa4f225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.050712 2949 raft_consensus.cc:1261] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Refusing update from remote peer 452c5e5bb2834c47b1f19845faa4f225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.051102 3308 consensus_queue.cc:1035] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.051247 3304 consensus_queue.cc:1035] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.053752 3311 mvcc.cc:205] Tried to move back new op lower bound from 7333273231564890112 to 7333273231530553344. Current Snapshot: MvccSnapshot[applied={T|T < 7333273231564890112}]
I20260925 15:25:10.054385 3312 mvcc.cc:205] Tried to move back new op lower bound from 7333273231564890112 to 7333273231530553344. Current Snapshot: MvccSnapshot[applied={T|T < 7333273231564890112}]
I20260925 15:25:10.057595 2791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:10.057706 2791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:10.058480 2791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Sending DeleteTablet for 3 replicas of tablet cf6529f9c9d84be09eb8576fce83a577
I20260925 15:25:10.058840 3069 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6529f9c9d84be09eb8576fce83a577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:35236
I20260925 15:25:10.058851 3209 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6529f9c9d84be09eb8576fce83a577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42238
I20260925 15:25:10.059103 3321 tablet_replica.cc:330] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: stopping tablet replica
I20260925 15:25:10.059090 2929 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6529f9c9d84be09eb8576fce83a577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:34370
I20260925 15:25:10.059229 3321 raft_consensus.cc:2229] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:10.059302 3323 tablet_replica.cc:330] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: stopping tablet replica
I20260925 15:25:10.059376 3321 raft_consensus.cc:2258] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.059381 3323 raft_consensus.cc:2229] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:10.059404 2791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:10.059547 2791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:10.059563 3323 raft_consensus.cc:2258] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.060000 3322 tablet_replica.cc:330] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: stopping tablet replica
I20260925 15:25:10.060084 3321 ts_tablet_manager.cc:1905] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.060196 3322 raft_consensus.cc:2229] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:10.060346 3322 raft_consensus.cc:2258] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.060850 3322 ts_tablet_manager.cc:1905] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.061432 3209 tablet_service.cc:1467] Processing CreateTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb (DEFAULT_TABLE table=test-table [id=8594a081e9374abb9434382970591e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:10.061414 2929 tablet_service.cc:1467] Processing CreateTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb (DEFAULT_TABLE table=test-table [id=8594a081e9374abb9434382970591e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:10.061549 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4287bc1cf47649dd8d1d5405bc681aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.061580 2929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4287bc1cf47649dd8d1d5405bc681aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.062139 3321 ts_tablet_manager.cc:1918] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:10.062217 3321 log.cc:1196] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/cf6529f9c9d84be09eb8576fce83a577
I20260925 15:25:10.062516 3321 ts_tablet_manager.cc:1939] T cf6529f9c9d84be09eb8576fce83a577 P 452c5e5bb2834c47b1f19845faa4f225: Deleting consensus metadata
I20260925 15:25:10.062948 3298 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Bootstrap starting.
I20260925 15:25:10.063383 3298 tablet_bootstrap.cc:654] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.063568 3322 ts_tablet_manager.cc:1918] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:10.063691 3322 log.cc:1196] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/cf6529f9c9d84be09eb8576fce83a577
I20260925 15:25:10.063685 3299 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Bootstrap starting.
I20260925 15:25:10.063982 3298 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: No bootstrap required, opened a new log
I20260925 15:25:10.064043 3298 ts_tablet_manager.cc:1397] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:10.064116 3322 ts_tablet_manager.cc:1939] T cf6529f9c9d84be09eb8576fce83a577 P 3461635441b04027b6ec30189df06d21: Deleting consensus metadata
I20260925 15:25:10.064214 3298 raft_consensus.cc:348] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.064321 3298 raft_consensus.cc:374] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.064363 3298 raft_consensus.cc:729] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452c5e5bb2834c47b1f19845faa4f225, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.064450 3299 tablet_bootstrap.cc:654] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.064404 3298 consensus_queue.cc:260] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.064581 3298 ts_tablet_manager.cc:1428] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:10.064977 2776 catalog_manager.cc:4841] TS 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985): tablet cf6529f9c9d84be09eb8576fce83a577 (table test-table [id=399349988bf2433b934921cffcebdafb]) successfully deleted
I20260925 15:25:10.065232 2777 catalog_manager.cc:4841] TS 3461635441b04027b6ec30189df06d21 (127.1.163.66:42457): tablet cf6529f9c9d84be09eb8576fce83a577 (table test-table [id=399349988bf2433b934921cffcebdafb]) successfully deleted
I20260925 15:25:10.065310 3069 tablet_service.cc:1467] Processing CreateTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb (DEFAULT_TABLE table=test-table [id=8594a081e9374abb9434382970591e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:10.065433 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4287bc1cf47649dd8d1d5405bc681aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.065660 3323 ts_tablet_manager.cc:1905] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.066803 3302 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Bootstrap starting.
I20260925 15:25:10.067418 3302 tablet_bootstrap.cc:654] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.067625 3299 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: No bootstrap required, opened a new log
I20260925 15:25:10.067695 3299 ts_tablet_manager.cc:1397] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 15:25:10.067873 3299 raft_consensus.cc:348] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.067956 3299 raft_consensus.cc:374] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.067986 3299 raft_consensus.cc:729] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa85b82d3f54f74b437948554d05024, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.068051 3299 consensus_queue.cc:260] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.068156 3299 ts_tablet_manager.cc:1428] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:10.068280 3323 ts_tablet_manager.cc:1918] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:10.068346 3323 log.cc:1196] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/cf6529f9c9d84be09eb8576fce83a577
I20260925 15:25:10.068622 3323 ts_tablet_manager.cc:1939] T cf6529f9c9d84be09eb8576fce83a577 P baa85b82d3f54f74b437948554d05024: Deleting consensus metadata
I20260925 15:25:10.069170 2779 catalog_manager.cc:4841] TS baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521): tablet cf6529f9c9d84be09eb8576fce83a577 (table test-table [id=399349988bf2433b934921cffcebdafb]) successfully deleted
I20260925 15:25:10.069367 3302 tablet_bootstrap.cc:492] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: No bootstrap required, opened a new log
I20260925 15:25:10.069424 3302 ts_tablet_manager.cc:1397] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:25:10.069594 3302 raft_consensus.cc:348] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.069698 3302 raft_consensus.cc:374] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.069721 3302 raft_consensus.cc:729] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3461635441b04027b6ec30189df06d21, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.069764 3302 consensus_queue.cc:260] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.069859 3302 ts_tablet_manager.cc:1428] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260925 15:25:10.078886 2996 tablet.cc:2369] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:25:10.181619 3276 tablet.cc:2369] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:10.309852 3306 raft_consensus.cc:482] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:10.309962 3306 raft_consensus.cc:504] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.310341 3306 leader_election.cc:288] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985)
I20260925 15:25:10.314886 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.310734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:58331 (local address 127.1.163.65:36521)
0925 15:25:10.310788 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.310790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.310991 (+ 201us) server_negotiation.cc:407] Connection header received
0925 15:25:10.311030 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.311031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.311041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.311098 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:10.311365 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.312117 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.313888 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.314476 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.314566 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.314575 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.314769 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.314814 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:10.314954 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.310719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:42331 (local address 127.1.163.67:39985)
0925 15:25:10.310793 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.310796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.310861 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:25:10.310995 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.310997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.311015 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.311076 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:10.311369 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.312161 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.313945 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.314497 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.314566 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.314575 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.314770 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.314813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:25:10.314985 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.310604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:36521 (local address 127.1.163.66:58331)
0925 15:25:10.310896 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.310906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.310976 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.311153 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.311154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.311159 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:10.311274 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.311276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.312262 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.312263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.313740 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.313744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.314593 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.314594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.314678 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.314684 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.314749 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.314808 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":149,"thread_start_us":61,"threads_started":1}
I20260925 15:25:10.314976 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.310604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:39985 (local address 127.1.163.66:42331)
0925 15:25:10.310783 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.310796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.310874 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.311153 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.311154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.311157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:10.311275 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.311276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.312261 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.312263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.313740 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.313744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.314591 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.314593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.314678 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.314683 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.314749 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.314809 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":90,"thread_start_us":70,"threads_started":1}
I20260925 15:25:10.315385 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4287bc1cf47649dd8d1d5405bc681aeb" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024" is_pre_election: true
I20260925 15:25:10.315488 2949 raft_consensus.cc:2454] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 0.
I20260925 15:25:10.315662 3023 leader_election.cc:302] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3461635441b04027b6ec30189df06d21, baa85b82d3f54f74b437948554d05024; no voters:
I20260925 15:25:10.315809 3306 raft_consensus.cc:2781] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:10.315873 3306 raft_consensus.cc:482] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:10.315896 3306 raft_consensus.cc:3037] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.315809 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4287bc1cf47649dd8d1d5405bc681aeb" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "452c5e5bb2834c47b1f19845faa4f225" is_pre_election: true
I20260925 15:25:10.315944 3229 raft_consensus.cc:2454] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 0.
I20260925 15:25:10.316717 3306 raft_consensus.cc:504] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.316844 3306 leader_election.cc:288] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 election: Requested vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985)
I20260925 15:25:10.317027 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4287bc1cf47649dd8d1d5405bc681aeb" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "452c5e5bb2834c47b1f19845faa4f225"
I20260925 15:25:10.317101 3229 raft_consensus.cc:3037] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.317076 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4287bc1cf47649dd8d1d5405bc681aeb" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024"
I20260925 15:25:10.317209 2949 raft_consensus.cc:3037] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.317813 3229 raft_consensus.cc:2454] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 1.
I20260925 15:25:10.317888 2949 raft_consensus.cc:2454] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 1.
I20260925 15:25:10.318058 3023 leader_election.cc:302] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3461635441b04027b6ec30189df06d21, baa85b82d3f54f74b437948554d05024; no voters:
I20260925 15:25:10.318215 3306 raft_consensus.cc:2781] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:10.318401 3306 raft_consensus.cc:686] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 1 LEADER]: Becoming Leader. State: Replica: 3461635441b04027b6ec30189df06d21, State: Running, Role: LEADER
I20260925 15:25:10.318493 3306 consensus_queue.cc:237] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.319252 2792 catalog_manager.cc:5495] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3461635441b04027b6ec30189df06d21 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:25:10 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:10.385300 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.377385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40445 (local address 127.0.0.1:46122)
0925 15:25:10.377713 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.377725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.377821 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.378045 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.378046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.378357 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:10.378403 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.378405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.379092 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.379093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.379645 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.379647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.380006 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.380007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.380065 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.380748 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:10.380753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:10.382248 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:10.384132 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:10.384140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:10.384142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:10.384352 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:10.384651 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:10.384652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:10.384652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:10.384711 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:10.385004 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:10.385010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:10.385055 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.385108 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.385201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260925 15:25:10.385350 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.377515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.1.163.126:40445)
0925 15:25:10.377695 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.377698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.377799 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:25:10.377951 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.377952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.377964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.378040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:10.378496 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.379008 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.379742 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.379924 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.380006 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.382360 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:10.382361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:10.382362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:10.382368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:10.384044 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.384446 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.384448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.384448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.384520 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.384802 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.384803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.384803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.384939 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:10.385004 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.385143 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.385251 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":148}
I20260925 15:25:10.387279 2792 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:46122:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:25:10.387462 2792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:10.389083 3209 tablet_service.cc:1467] Processing CreateTablet for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b0069704a7f470a866377d0080bb3ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:10.389078 3069 tablet_service.cc:1467] Processing CreateTablet for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b0069704a7f470a866377d0080bb3ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:10.389091 2929 tablet_service.cc:1467] Processing CreateTablet for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b0069704a7f470a866377d0080bb3ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:10.389249 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.389248 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.389263 2929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b70ae0d4d3b4b9d9bd0cfeaacf91332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:10.390583 3299 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: Bootstrap starting.
I20260925 15:25:10.390719 3298 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: Bootstrap starting.
I20260925 15:25:10.390859 3302 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: Bootstrap starting.
I20260925 15:25:10.391017 3299 tablet_bootstrap.cc:654] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.391072 3298 tablet_bootstrap.cc:654] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.391288 3302 tablet_bootstrap.cc:654] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:10.391721 3298 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: No bootstrap required, opened a new log
I20260925 15:25:10.391721 3299 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: No bootstrap required, opened a new log
I20260925 15:25:10.391783 3298 ts_tablet_manager.cc:1397] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:10.391782 3299 ts_tablet_manager.cc:1397] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:10.391907 3298 raft_consensus.cc:348] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.391950 3299 raft_consensus.cc:348] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.392005 3298 raft_consensus.cc:374] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.392030 3302 tablet_bootstrap.cc:492] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: No bootstrap required, opened a new log
I20260925 15:25:10.392038 3298 raft_consensus.cc:729] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452c5e5bb2834c47b1f19845faa4f225, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.392022 3299 raft_consensus.cc:374] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.392091 3299 raft_consensus.cc:729] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa85b82d3f54f74b437948554d05024, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.392097 3302 ts_tablet_manager.cc:1397] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:10.392082 3298 consensus_queue.cc:260] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.392151 3299 consensus_queue.cc:260] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.392180 3298 ts_tablet_manager.cc:1428] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:10.392253 3299 ts_tablet_manager.cc:1428] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:10.392246 3302 raft_consensus.cc:348] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.392275 3318 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.392331 3302 raft_consensus.cc:374] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:10.392360 3302 raft_consensus.cc:729] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3461635441b04027b6ec30189df06d21, State: Initialized, Role: FOLLOWER
I20260925 15:25:10.392354 3318 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.392418 3302 consensus_queue.cc:260] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.392431 3303 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.392517 3302 ts_tablet_manager.cc:1428] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:10.392505 3303 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.392701 3306 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.392807 3306 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.616648 3306 raft_consensus.cc:482] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:10.616755 3306 raft_consensus.cc:504] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.616983 3306 leader_election.cc:288] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985)
I20260925 15:25:10.617300 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b70ae0d4d3b4b9d9bd0cfeaacf91332" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024" is_pre_election: true
I20260925 15:25:10.617300 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b70ae0d4d3b4b9d9bd0cfeaacf91332" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "452c5e5bb2834c47b1f19845faa4f225" is_pre_election: true
I20260925 15:25:10.617401 2949 raft_consensus.cc:2454] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 0.
I20260925 15:25:10.617401 3229 raft_consensus.cc:2454] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 0.
I20260925 15:25:10.617591 3020 leader_election.cc:302] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3461635441b04027b6ec30189df06d21, 452c5e5bb2834c47b1f19845faa4f225; no voters:
I20260925 15:25:10.617730 3306 raft_consensus.cc:2781] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:10.617779 3306 raft_consensus.cc:482] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:10.617817 3306 raft_consensus.cc:3037] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.618600 3306 raft_consensus.cc:504] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.618754 3306 leader_election.cc:288] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 election: Requested vote from peers baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521), 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985)
I20260925 15:25:10.618909 2949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b70ae0d4d3b4b9d9bd0cfeaacf91332" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa85b82d3f54f74b437948554d05024"
I20260925 15:25:10.619004 2949 raft_consensus.cc:3037] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.618984 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b70ae0d4d3b4b9d9bd0cfeaacf91332" candidate_uuid: "3461635441b04027b6ec30189df06d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "452c5e5bb2834c47b1f19845faa4f225"
I20260925 15:25:10.619057 3229 raft_consensus.cc:3037] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:10.619799 3229 raft_consensus.cc:2454] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 1.
I20260925 15:25:10.619799 2949 raft_consensus.cc:2454] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3461635441b04027b6ec30189df06d21 in term 1.
I20260925 15:25:10.620016 3020 leader_election.cc:302] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3461635441b04027b6ec30189df06d21, 452c5e5bb2834c47b1f19845faa4f225; no voters:
I20260925 15:25:10.620131 3306 raft_consensus.cc:2781] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:10.620222 3306 raft_consensus.cc:686] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [term 1 LEADER]: Becoming Leader. State: Replica: 3461635441b04027b6ec30189df06d21, State: Running, Role: LEADER
I20260925 15:25:10.620286 3306 consensus_queue.cc:237] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } }
I20260925 15:25:10.620602 3329 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: TxnStatusTablet state changed. Reason: New leader 3461635441b04027b6ec30189df06d21. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.620689 3329 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:10.620926 3341 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:25:10.621022 3341 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:25:10.620960 2792 catalog_manager.cc:5495] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3461635441b04027b6ec30189df06d21 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:10.682725 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.679309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.1.163.66:42457)
0925 15:25:10.679419 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.679421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.679429 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:10.679483 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.679484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.679518 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.679913 (+ 395us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:10.680060 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.680624 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.681493 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.681859 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.681952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.682206 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.682340 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.682443 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.682599 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.682629 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260925 15:25:10.682691 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.679119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42457 (local address 127.0.0.1:35244)
0925 15:25:10.679311 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.679327 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.679386 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.679920 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.679921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.679923 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:10.679984 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.679986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.680717 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.680719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.681383 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.681386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.681976 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.681977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.682071 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.682089 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.682518 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.682527 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.682570 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.682619 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 15:25:10.684111 2949 raft_consensus.cc:1261] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Refusing update from remote peer 3461635441b04027b6ec30189df06d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.684157 3229 raft_consensus.cc:1261] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Refusing update from remote peer 3461635441b04027b6ec30189df06d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.684386 3329 consensus_queue.cc:1035] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.684496 3306 consensus_queue.cc:1035] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.685114 3303 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: TxnStatusTablet state changed. Reason: New leader 3461635441b04027b6ec30189df06d21. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.685222 3303 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.685495 3318 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: TxnStatusTablet state changed. Reason: New leader 3461635441b04027b6ec30189df06d21. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.685587 3318 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.685786 3345 mvcc.cc:205] Tried to move back new op lower bound from 7333273234159775744 to 7333273233900929024. Current Snapshot: MvccSnapshot[applied={T|T < 7333273234159775744}]
I20260925 15:25:10.685767 3306 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.685859 3306 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:10.686633 3303 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.686743 3303 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P baa85b82d3f54f74b437948554d05024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.687117 3329 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.687211 3329 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 3461635441b04027b6ec30189df06d21: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:10.687673 3318 tablet_replica.cc:439] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3461635441b04027b6ec30189df06d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 } } peers { permanent_uuid: "3461635441b04027b6ec30189df06d21" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 42457 } } peers { permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 } } }
I20260925 15:25:10.687769 3318 tablet_replica.cc:442] T 2b70ae0d4d3b4b9d9bd0cfeaacf91332 P 452c5e5bb2834c47b1f19845faa4f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:10.692169 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.688703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.1.163.66:42457)
0925 15:25:10.688741 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.688743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.689564 (+ 821us) server_negotiation.cc:407] Connection header received
0925 15:25:10.689608 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.689609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.689619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.689671 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:10.689807 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.690301 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.691351 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.691575 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.691638 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.691835 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.691912 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.691970 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.692093 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.692116 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:25:10.698760 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.695010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:49065 (local address 127.1.163.66:42457)
0925 15:25:10.695054 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.695056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.695189 (+ 133us) server_negotiation.cc:407] Connection header received
0925 15:25:10.695216 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.695217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.695227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.695292 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:10.695387 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.696089 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.697870 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.698209 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.698292 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.698496 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.698530 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.698588 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.698634 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.698658 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:10.698782 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.694944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:42457 (local address 127.1.163.66:49065)
0925 15:25:10.695106 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:10.695117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:10.695179 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:10.695297 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:10.695297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:10.695299 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:10.695345 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.695346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.696191 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.696192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.697770 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:10.697773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.698301 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.698302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:10.698357 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.698370 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:10.698589 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:10.698591 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:25:10.698630 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 15:25:10.698667 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260925 15:25:10.701285 3229 raft_consensus.cc:1261] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Refusing update from remote peer 3461635441b04027b6ec30189df06d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.701308 2949 raft_consensus.cc:1261] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Refusing update from remote peer 3461635441b04027b6ec30189df06d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:10.701655 3306 consensus_queue.cc:1035] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa85b82d3f54f74b437948554d05024" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.701807 3329 consensus_queue.cc:1035] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "452c5e5bb2834c47b1f19845faa4f225" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:10.716830 2791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:10.716951 2791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:10.717636 2791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae59610434d4f8c8d365d99d30a2719: Sending DeleteTablet for 3 replicas of tablet 4287bc1cf47649dd8d1d5405bc681aeb
I20260925 15:25:10.717880 2929 tablet_service.cc:1514] Processing DeleteTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:34370
I20260925 15:25:10.717904 3209 tablet_service.cc:1514] Processing DeleteTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42238
I20260925 15:25:10.718022 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet 4287bc1cf47649dd8d1d5405bc681aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:10 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:35236
I20260925 15:25:10.718210 3365 tablet_replica.cc:330] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: stopping tablet replica
I20260925 15:25:10.718268 3365 raft_consensus.cc:2229] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:10.718356 3365 raft_consensus.cc:2258] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.718667 3365 ts_tablet_manager.cc:1905] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.718843 3364 tablet_replica.cc:330] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: stopping tablet replica
I20260925 15:25:10.718911 3364 raft_consensus.cc:2229] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:10.718950 3364 raft_consensus.cc:2258] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.719208 3364 ts_tablet_manager.cc:1905] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.720669 3363 tablet_replica.cc:330] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: stopping tablet replica
I20260925 15:25:10.720788 3365 ts_tablet_manager.cc:1918] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:10.720796 3363 raft_consensus.cc:2229] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:10.720857 3365 log.cc:1196] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/4287bc1cf47649dd8d1d5405bc681aeb
I20260925 15:25:10.720975 3363 raft_consensus.cc:2258] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:10.721163 3365 ts_tablet_manager.cc:1939] T 4287bc1cf47649dd8d1d5405bc681aeb P 3461635441b04027b6ec30189df06d21: Deleting consensus metadata
I20260925 15:25:10.721076 3364 ts_tablet_manager.cc:1918] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:10.721252 3364 log.cc:1196] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/4287bc1cf47649dd8d1d5405bc681aeb
I20260925 15:25:10.721413 3363 ts_tablet_manager.cc:1905] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:10.721490 3364 ts_tablet_manager.cc:1939] T 4287bc1cf47649dd8d1d5405bc681aeb P 452c5e5bb2834c47b1f19845faa4f225: Deleting consensus metadata
I20260925 15:25:10.721509 2777 catalog_manager.cc:4841] TS 3461635441b04027b6ec30189df06d21 (127.1.163.66:42457): tablet 4287bc1cf47649dd8d1d5405bc681aeb (table test-table [id=8594a081e9374abb9434382970591e7f]) successfully deleted
I20260925 15:25:10.721786 2776 catalog_manager.cc:4841] TS 452c5e5bb2834c47b1f19845faa4f225 (127.1.163.67:39985): tablet 4287bc1cf47649dd8d1d5405bc681aeb (table test-table [id=8594a081e9374abb9434382970591e7f]) successfully deleted
I20260925 15:25:10.723263 3363 ts_tablet_manager.cc:1918] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:10.723336 3363 log.cc:1196] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/4287bc1cf47649dd8d1d5405bc681aeb
I20260925 15:25:10.723568 3363 ts_tablet_manager.cc:1939] T 4287bc1cf47649dd8d1d5405bc681aeb P baa85b82d3f54f74b437948554d05024: Deleting consensus metadata
I20260925 15:25:10.723971 2779 catalog_manager.cc:4841] TS baa85b82d3f54f74b437948554d05024 (127.1.163.65:36521): tablet 4287bc1cf47649dd8d1d5405bc681aeb (table test-table [id=8594a081e9374abb9434382970591e7f]) successfully deleted
Sep 25 15:25:10 dist-test-slave-hvv5 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:10.730650 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.719095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.1.163.65:36521)
0925 15:25:10.719137 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.719139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.720646 (+ 1507us) server_negotiation.cc:407] Connection header received
0925 15:25:10.721736 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.721737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.721750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.721783 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:10.722430 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.723016 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.723947 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.724223 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.724327 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.727153 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:10.727163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:10.727164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:10.727172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:10.729333 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.729713 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.729716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.729717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.729800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.729999 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.730000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.730000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.730181 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:10.730346 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.730467 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.730557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:25:10.730873 2985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34384
I20260925 15:25:10.738698 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:10.731905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.1.163.126:40445)
0925 15:25:10.731968 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:10.731969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:10.732086 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:25:10.732165 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:10.732166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:10.732178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:10.732229 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:10.732901 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.733499 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.734283 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:10.734499 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:10.734586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:10.735828 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:10.735829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:10.735830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:10.735837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:10.737482 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.737876 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.737878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.737878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.737944 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:10.738167 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:10.738168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:10.738169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:10.738298 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:10.738409 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:10.738509 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:25:10.738589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:25:10.738947 2791 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46138
I20260925 15:25:10.739642 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2864
I20260925 15:25:10.745784 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3004
I20260925 15:25:10.751693 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3144
I20260925 15:25:10.757510 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 2760
2026-09-25T15:25:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2727 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:10 dist-test-slave-hvv5 krb5kdc[3372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:10 dist-test-slave-hvv5 krb5kdc[3372](info): set up 2 sockets
Sep 25 15:25:10 dist-test-slave-hvv5 krb5kdc[3372](info): commencing operation
krb5kdc: starting...
W20260925 15:25:11.808393 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:11 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:11.940264 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44617
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:12.023720 3385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:12.023968 3385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:12.023995 3385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:12.025501 3385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:12.025557 3385 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:25:12.025573 3385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:12.025586 3385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:12.025610 3385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:12.025674 3385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:12.027406 3385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44617
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:12.027691 3385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:12.027959 3385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:12.032958 3393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.032967 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.033002 3385 server_base.cc:1034] running on GCE node
W20260925 15:25:12.032987 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.033659 3385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:12.033985 3385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:12.035161 3385 hybrid_clock.cc:648] HybridClock initialized: now 1790349912035142 us; error 28 us; skew 500 ppm
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:12.037925 3385 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:12.038408 3385 webserver.cc:460] Webserver started at http://127.1.163.126:41501/ using document root <none> and password file <none>
I20260925 15:25:12.038646 3385 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:12.038714 3385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:12.038868 3385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:12.039947 3385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "5efb3c732a2c4edf88e218088bf01540"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "a94406a3185d527792219124d0200e06"
server_key_iv: "547ba8095e1cefe75e66ada9d447c555"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.040309 3385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "5efb3c732a2c4edf88e218088bf01540"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "a94406a3185d527792219124d0200e06"
server_key_iv: "547ba8095e1cefe75e66ada9d447c555"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.041818 3385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.042725 3400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.042923 3385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.043052 3385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "5efb3c732a2c4edf88e218088bf01540"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "a94406a3185d527792219124d0200e06"
server_key_iv: "547ba8095e1cefe75e66ada9d447c555"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.043143 3385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:12.057438 3385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:12.059490 3385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:12.059659 3385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:12.064426 3385 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44617
I20260925 15:25:12.064471 3462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44617 every 8 connection(s)
I20260925 15:25:12.064843 3385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:12.065351 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3385
I20260925 15:25:12.065421 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:12.065536 3463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.065604 1677 external_mini_cluster.cc:1428] Setting key 836e2c893277785db80bbb0efa0a242c
I20260925 15:25:12.068405 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Bootstrap starting.
I20260925 15:25:12.069059 3463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.069339 3463 log.cc:824] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:12.070165 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: No bootstrap required, opened a new log
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.072889 3463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } }
I20260925 15:25:12.073082 3463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.073110 3463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efb3c732a2c4edf88e218088bf01540, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.073268 3463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } }
I20260925 15:25:12.073328 3463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:12.073349 3463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:12.073419 3463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:12.073982 3463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } }
I20260925 15:25:12.074146 3463 leader_election.cc:302] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efb3c732a2c4edf88e218088bf01540; no voters:
I20260925 15:25:12.074370 3463 leader_election.cc:288] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:12.074404 3468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:12.074611 3468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [term 1 LEADER]: Becoming Leader. State: Replica: 5efb3c732a2c4edf88e218088bf01540, State: Running, Role: LEADER
I20260925 15:25:12.074651 3463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:12.074805 3468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } }
I20260925 15:25:12.075187 3470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5efb3c732a2c4edf88e218088bf01540. Latest consensus state: current_term: 1 leader_uuid: "5efb3c732a2c4edf88e218088bf01540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } } }
I20260925 15:25:12.075285 3470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:12.075507 3477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:12.076066 3477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:12.076975 3469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5efb3c732a2c4edf88e218088bf01540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efb3c732a2c4edf88e218088bf01540" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44617 } } }
I20260925 15:25:12.077059 3469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:12.078961 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.066161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.1.163.126:44617)
0925 15:25:12.066354 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.066362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.066387 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:25:12.066991 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.066999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.067049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.067206 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.067835 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.068828 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.069732 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.070345 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.070470 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.073558 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.073562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.073562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.073581 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.075591 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.076035 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.076039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.076039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.076143 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.076470 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.076471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.076471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.076767 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.076944 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.078613 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.078764 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260925 15:25:12.079952 3477 catalog_manager.cc:1347] Generated new cluster ID: c8d2f20c8cc94c3fb44ede853a15148f
I20260925 15:25:12.080008 3477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:12.103137 3477 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:12.103895 3477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:12.112887 3477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Generated new TSK 0
I20260925 15:25:12.113124 3477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:12.149186 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44617
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:12.235749 3489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:12.236011 3489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:12.236055 3489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:12.237360 3489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:12.237430 3489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:12.237478 3489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:25:12.237504 3489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:12.237540 3489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:12.239108 3489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:12.239379 3489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:12.239616 3489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:12.244697 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.244705 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.244663 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.244817 3489 server_base.cc:1034] running on GCE node
I20260925 15:25:12.245113 3489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:12.245363 3489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:12.246517 3489 hybrid_clock.cc:648] HybridClock initialized: now 1790349912246486 us; error 36 us; skew 500 ppm
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:12.249187 3489 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:12.249641 3489 webserver.cc:460] Webserver started at http://127.1.163.65:38465/ using document root <none> and password file <none>
I20260925 15:25:12.249874 3489 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:12.249919 3489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:12.250016 3489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:12.251048 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "f1e291f1fb77455194b8b00873c9a9a5"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "22f9ae4d067c2d4170a0ad64fb30b975"
server_key_iv: "7ef166e2e4cfaccdef0b7e9d9871dbd8"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.251355 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "f1e291f1fb77455194b8b00873c9a9a5"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "22f9ae4d067c2d4170a0ad64fb30b975"
server_key_iv: "7ef166e2e4cfaccdef0b7e9d9871dbd8"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.252676 3489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:12.253494 3504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.253804 3489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.253894 3489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "f1e291f1fb77455194b8b00873c9a9a5"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "22f9ae4d067c2d4170a0ad64fb30b975"
server_key_iv: "7ef166e2e4cfaccdef0b7e9d9871dbd8"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.253958 3489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:12.267943 3489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:12.270098 3489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:12.270226 3489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:12.270823 3489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:12.270889 3489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.270926 3489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:12.270957 3489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.279091 3489 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:33263
I20260925 15:25:12.279165 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:33263 every 8 connection(s)
I20260925 15:25:12.279570 3489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:12.281690 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.271272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:58243 (local address 127.1.163.126:44617)
0925 15:25:12.271335 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.271337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.271585 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:25:12.272161 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.272163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.272179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.272255 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.273035 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.273552 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.274829 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.275174 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.275262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.278320 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.278322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.278322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.278331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.280189 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.280670 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.280672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.280672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.280737 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.280999 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.281000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.281001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.281127 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.281205 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.281497 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.281581 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:25:12.281890 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.271262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.65:58243)
0925 15:25:12.271454 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.271490 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.272045 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.272348 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.272352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.272702 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.272895 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.272900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.273660 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.273661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.274657 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.274661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.275273 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.275274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.275455 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.276047 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.276057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.278204 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.280289 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.280296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.280298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.280590 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.280804 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.280805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.280806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.280925 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.281194 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.281201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.281349 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.281462 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.281631 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260925 15:25:12.282387 3620 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44617
I20260925 15:25:12.282470 3620 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:12.282656 3620 heartbeater.cc:511] Master 127.1.163.126:44617 requested a full tablet report, sending...
I20260925 15:25:12.283252 3416 ts_manager.cc:194] Registered new tserver with Master: f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263)
I20260925 15:25:12.284051 3416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:58243
I20260925 15:25:12.285249 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3489
I20260925 15:25:12.285358 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:12.285543 1677 external_mini_cluster.cc:1428] Setting key 08d384672c56076b5a8a874ed11a935f
I20260925 15:25:12.291064 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.284263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:39439 (local address 127.1.163.126:44617)
0925 15:25:12.284304 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.284306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.284320 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:25:12.284340 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.284340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.284349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.284434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.284851 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.285333 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.286314 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.286620 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.286703 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.287880 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.287882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.287882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.287890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.289505 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.289989 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.289991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.289992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.290109 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.290441 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.290442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.290443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.290574 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.290637 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.290910 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.290986 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:12.291132 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.283425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.65:39439)
0925 15:25:12.283570 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.283582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.283637 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.284457 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.284458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.284700 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.284753 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.284754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.285465 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.285466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.286225 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.286228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.286758 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.286759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.286852 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.287442 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.287448 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.287761 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.289602 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.289603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.289604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.289889 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.290189 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.290190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.290190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.290316 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.290658 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.290659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.290724 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.290814 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.290955 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:12.316722 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44617
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:12.403208 3629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:12.403461 3629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:12.403491 3629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:12.404891 3629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:12.404950 3629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:12.404973 3629 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:25:12.404985 3629 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:12.405036 3629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:12.406574 3629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:12.406852 3629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:12.407109 3629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:12.412822 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.412883 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.412848 3629 server_base.cc:1034] running on GCE node
W20260925 15:25:12.412829 3637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.413298 3629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:12.413585 3629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:12.414763 3629 hybrid_clock.cc:648] HybridClock initialized: now 1790349912414743 us; error 39 us; skew 500 ppm
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:12.422585 3629 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:12.423125 3629 webserver.cc:460] Webserver started at http://127.1.163.66:44843/ using document root <none> and password file <none>
I20260925 15:25:12.423373 3629 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:12.423450 3629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:12.423610 3629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:12.424786 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "f6991e6957d14551bf3519aea29e6e63"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "46ac05f8c45aa68ec233ba58d9afc9e0"
server_key_iv: "761d1c65b36cb992ab828a6f2302a701"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.425163 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "f6991e6957d14551bf3519aea29e6e63"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "46ac05f8c45aa68ec233ba58d9afc9e0"
server_key_iv: "761d1c65b36cb992ab828a6f2302a701"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.426611 3629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:12.427489 3644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.427704 3629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.427815 3629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "f6991e6957d14551bf3519aea29e6e63"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "46ac05f8c45aa68ec233ba58d9afc9e0"
server_key_iv: "761d1c65b36cb992ab828a6f2302a701"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.427893 3629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:12.444408 3629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:12.447160 3629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:12.447336 3629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:12.447901 3629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:12.447973 3629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.448022 3629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:12.448046 3629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.455427 3629 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34659
I20260925 15:25:12.455519 3759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34659 every 8 connection(s)
I20260925 15:25:12.455856 3629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.459607 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.448481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:47107 (local address 127.1.163.126:44617)
0925 15:25:12.448527 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.448529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.448883 (+ 354us) server_negotiation.cc:407] Connection header received
0925 15:25:12.449551 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.449552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.449567 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.449652 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.450660 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.451520 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.453376 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.453602 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.453666 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.456631 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.456632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.456633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.456641 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.458191 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.458615 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.458616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.458616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.458678 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.458933 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.458935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.458935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.459058 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.459131 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.459435 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.459516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:12.459803 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.448498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.66:47107)
0925 15:25:12.448715 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.448754 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.449426 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.449769 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.449774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.450213 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.450544 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.450553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.451641 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.451642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.453250 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.453254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.453703 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.453705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.453888 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.454623 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.454644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.456506 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.458285 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.458293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.458295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.458527 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.458747 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.458748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.458748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.458860 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.459134 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.459141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.459287 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.459392 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.459563 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260925 15:25:12.460310 3760 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44617
I20260925 15:25:12.460392 3760 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:12.460625 3760 heartbeater.cc:511] Master 127.1.163.126:44617 requested a full tablet report, sending...
I20260925 15:25:12.460968 3417 ts_manager.cc:194] Registered new tserver with Master: f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:12.461369 3417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:47107
I20260925 15:25:12.462545 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3629
I20260925 15:25:12.462643 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:12.462818 1677 external_mini_cluster.cc:1428] Setting key 6c862fd2ee708ca4e8199072f385e3ca
I20260925 15:25:12.468669 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.461482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40595 (local address 127.1.163.126:44617)
0925 15:25:12.461521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.461523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.461691 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:25:12.461754 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.461755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.461772 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.461812 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.462205 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.463133 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.463961 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.464277 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.464350 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.465494 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.465495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.465495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.465501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.467344 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.467742 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.467743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.467744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.467808 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.468034 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.468036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.468036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.468197 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.468277 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.468476 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.468587 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:12.468739 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.461414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.66:40595)
0925 15:25:12.461618 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.461628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.461679 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.461828 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.461829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.462038 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.462127 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.462129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.463237 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.463238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.463877 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.463879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.464372 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.464373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.464434 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.464958 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.464963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.465387 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.467445 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.467447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.467447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.467621 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.467896 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.467897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.467898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.467959 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.468282 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.468283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.468335 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.468416 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.468581 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:12.493396 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44617
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:12.583019 3769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:12.583257 3769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:12.583288 3769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:12.584841 3769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:12.584944 3769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:12.584969 3769 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:25:12.585003 3769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:12.585036 3769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:12.586891 3769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:12.587170 3769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:12.587435 3769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:12.593021 3777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.593021 3775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:12.593057 3774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:12.593119 3769 server_base.cc:1034] running on GCE node
I20260925 15:25:12.593659 3769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:12.593942 3769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:12.595124 3769 hybrid_clock.cc:648] HybridClock initialized: now 1790349912595114 us; error 57 us; skew 500 ppm
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:12.598109 3769 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:12.598645 3769 webserver.cc:460] Webserver started at http://127.1.163.67:38721/ using document root <none> and password file <none>
I20260925 15:25:12.598893 3769 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:12.598963 3769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:12.599116 3769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:12.600196 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "f48030e187f14316b8a901cb012eed43"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "2397f21eead9a34cbd1c89c6eee298b1"
server_key_iv: "780e3c1f161cb61ff13c91b9b94f8852"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.600644 3769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "f48030e187f14316b8a901cb012eed43"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "2397f21eead9a34cbd1c89c6eee298b1"
server_key_iv: "780e3c1f161cb61ff13c91b9b94f8852"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.602337 3769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:12.603395 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.603622 3769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.603757 3769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "f48030e187f14316b8a901cb012eed43"
format_stamp: "Formatted at 2026-09-25 15:25:12 on dist-test-slave-hvv5"
server_key: "2397f21eead9a34cbd1c89c6eee298b1"
server_key_iv: "780e3c1f161cb61ff13c91b9b94f8852"
server_key_version: "encryptionkey@0"
I20260925 15:25:12.603886 3769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:12.629470 3769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:12.632035 3769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:12.632211 3769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:12.632881 3769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:12.632936 3769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.632988 3769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:12.633016 3769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.640681 3769 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:41187
I20260925 15:25:12.640780 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:41187 every 8 connection(s)
I20260925 15:25:12.641124 3769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:12.643839 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.633398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:38193 (local address 127.1.163.126:44617)
0925 15:25:12.633472 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.633474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.633741 (+ 267us) server_negotiation.cc:407] Connection header received
0925 15:25:12.634520 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.634522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.634539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.634633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.635617 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.636242 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.637465 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.637703 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.637811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.640558 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.640560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.640560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.640571 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.642287 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.642759 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.642760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.642761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.642836 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.643111 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.643112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.643112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.643252 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.643345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.643645 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.643736 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:12.644053 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.633373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.67:38193)
0925 15:25:12.633579 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.633638 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.634378 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.634715 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.634720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.635162 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.635500 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.635507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.636370 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.636372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.637358 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.637361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.637837 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.637838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.637975 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.638646 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.638656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.640408 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.642402 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.642410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.642412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.642655 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.642911 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.642912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.642913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.643034 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.643333 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.643341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.643497 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.643620 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.643810 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260925 15:25:12.644577 3900 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44617
I20260925 15:25:12.644667 3900 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:12.644881 3900 heartbeater.cc:511] Master 127.1.163.126:44617 requested a full tablet report, sending...
I20260925 15:25:12.645316 3417 ts_manager.cc:194] Registered new tserver with Master: f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:12.645840 3417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:38193
I20260925 15:25:12.649408 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 3769
I20260925 15:25:12.649516 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:12.649718 1677 external_mini_cluster.cc:1428] Setting key 09bdd834c0f389669736a3ecc4c8b29b
I20260925 15:25:12.650693 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:12.652736 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.645795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.1.163.67:36559)
0925 15:25:12.645970 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.645982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.646048 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.646225 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.646226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.646399 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.646462 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.646464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.647300 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.647301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.647880 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.647882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.648238 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.648238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.648296 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.649139 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.649144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.649446 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.651506 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.651507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.651507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.651678 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.651932 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.651933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.651934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.651997 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.652375 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.652376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.652419 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.652477 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.652636 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260925 15:25:12.652792 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.645855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:36559 (local address 127.1.163.126:44617)
0925 15:25:12.645902 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.645904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.646050 (+ 146us) server_negotiation.cc:407] Connection header received
0925 15:25:12.646114 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.646115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.646124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.646200 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.646550 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.647198 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.647978 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.648163 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.648230 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.649577 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.649578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.649579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.649585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.651403 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.651759 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.651760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.651761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.651834 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.652115 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.652116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.652116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.652298 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.652381 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.652520 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.652664 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.677668 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.670113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.1.163.126:44617)
0925 15:25:12.670202 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.670204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.670214 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:12.670240 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.670241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.670250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.670310 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.670952 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.671502 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.672269 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.672470 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.672525 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.675008 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.675010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.675010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.675018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.676464 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.676865 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.676866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.676866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.676922 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.677173 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.677174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.677174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.677284 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.677362 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.677502 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.677566 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:25:12.680737 3417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:12.682780 3417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:12.689843 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:41187 (local address 127.0.0.1:57902)
0925 15:25:12.685652 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.685663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.685719 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.686083 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.686084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.686086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:12.686145 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.686147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687500 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.687501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.688710 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.688713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689498 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.689581 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689597 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.689670 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.689732 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260925 15:25:12.689916 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.1.163.66:34659)
0925 15:25:12.685607 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.685612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.685624 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:12.685668 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.685670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.685688 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.685776 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:12.686106 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687005 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.688870 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689563 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689633 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689691 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.689761 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.689804 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260925 15:25:12.689843 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34659 (local address 127.0.0.1:47074)
0925 15:25:12.685467 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.685512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.685580 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.685799 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.685802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.685804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:12.685909 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.685913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687085 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.687086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.688711 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.688732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689636 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.689696 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689699 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.689750 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.689783 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":7,"thread_start_us":50,"threads_started":1}
I20260925 15:25:12.690310 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.1.163.67:41187)
0925 15:25:12.685888 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.685891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.685899 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:12.685953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.685955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.685971 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.686220 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:12.686281 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687389 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.688822 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689375 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689496 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689569 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.690126 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.690183 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":98,"threads_started":1}
I20260925 15:25:12.690388 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.1.163.65:33263)
0925 15:25:12.686011 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.686015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.686074 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:25:12.686195 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.686198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.686214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.686286 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:12.686464 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687556 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.688751 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689552 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689638 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689691 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.689945 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.690255 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":181,"threads_started":1}
I20260925 15:25:12.690730 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.685515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33263 (local address 127.0.0.1:54450)
0925 15:25:12.686019 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.686030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.686084 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.686349 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.686350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.686352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:12.686398 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.686400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.687637 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.687638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.688666 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.688669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.689723 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.689724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.689787 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.689792 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.689843 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.689892 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260925 15:25:12.691012 3694 tablet_service.cc:1467] Processing CreateTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 (DEFAULT_TABLE table=test-table [id=0386e9300e724f25b89feb368a9563de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:12.691006 3834 tablet_service.cc:1467] Processing CreateTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 (DEFAULT_TABLE table=test-table [id=0386e9300e724f25b89feb368a9563de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:12.691416 3694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f94ff684fd47a7bb67bbe05be831f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.691411 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f94ff684fd47a7bb67bbe05be831f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.691370 3554 tablet_service.cc:1467] Processing CreateTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 (DEFAULT_TABLE table=test-table [id=0386e9300e724f25b89feb368a9563de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:12.691687 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f94ff684fd47a7bb67bbe05be831f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.694682 3923 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Bootstrap starting.
I20260925 15:25:12.695134 3924 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Bootstrap starting.
I20260925 15:25:12.695405 3923 tablet_bootstrap.cc:654] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.695713 3923 log.cc:824] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:12.696175 3924 tablet_bootstrap.cc:654] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.696487 3923 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: No bootstrap required, opened a new log
I20260925 15:25:12.696503 3924 log.cc:824] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:12.696588 3923 ts_tablet_manager.cc:1397] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:12.696794 3925 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Bootstrap starting.
I20260925 15:25:12.697355 3925 tablet_bootstrap.cc:654] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.697588 3925 log.cc:824] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:12.698287 3925 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: No bootstrap required, opened a new log
I20260925 15:25:12.698359 3925 ts_tablet_manager.cc:1397] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:12.698652 3923 raft_consensus.cc:348] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.698815 3923 raft_consensus.cc:374] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.698863 3923 raft_consensus.cc:729] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.699018 3923 consensus_queue.cc:260] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.699308 3923 ts_tablet_manager.cc:1428] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:12.699352 3760 heartbeater.cc:503] Master 127.1.163.126:44617 was elected leader, sending a full tablet report...
I20260925 15:25:12.700068 3924 tablet_bootstrap.cc:492] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: No bootstrap required, opened a new log
I20260925 15:25:12.700203 3924 ts_tablet_manager.cc:1397] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:25:12.700393 3925 raft_consensus.cc:348] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.700587 3925 raft_consensus.cc:374] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.700629 3925 raft_consensus.cc:729] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e291f1fb77455194b8b00873c9a9a5, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.700755 3925 consensus_queue.cc:260] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.701030 3925 ts_tablet_manager.cc:1428] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:12.701097 3620 heartbeater.cc:503] Master 127.1.163.126:44617 was elected leader, sending a full tablet report...
I20260925 15:25:12.702250 3924 raft_consensus.cc:348] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.702383 3924 raft_consensus.cc:374] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.702405 3924 raft_consensus.cc:729] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48030e187f14316b8a901cb012eed43, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.702526 3924 consensus_queue.cc:260] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.702808 3924 ts_tablet_manager.cc:1428] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:12.702930 3900 heartbeater.cc:503] Master 127.1.163.126:44617 was elected leader, sending a full tablet report...
W20260925 15:25:12.706458 3761 tablet.cc:2369] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:25:12.780520 3621 tablet.cc:2369] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:12.850327 3931 raft_consensus.cc:482] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:12.850481 3931 raft_consensus.cc:504] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.850901 3931 leader_election.cc:288] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:12.855331 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.851286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:47203 (local address 127.1.163.65:33263)
0925 15:25:12.851330 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.851332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.851342 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:12.851383 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.851383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.851392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.851450 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:12.851687 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.852469 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854157 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.854797 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854908 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.854923 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.855204 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.855257 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:12.855331 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.851282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40181 (local address 127.1.163.66:34659)
0925 15:25:12.851330 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.851332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.851516 (+ 184us) server_negotiation.cc:407] Connection header received
0925 15:25:12.851574 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.851575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.851583 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.851627 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:12.851813 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.852508 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854157 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.854797 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854950 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.854961 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.855205 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.855256 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:12.855347 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.851142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33263 (local address 127.1.163.67:47203)
0925 15:25:12.851230 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.851246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.851309 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.851481 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.851481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.851483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:12.851567 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.851569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.852628 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.852630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.854017 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.854020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.854975 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.855095 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.855098 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.855164 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.855215 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":80}
I20260925 15:25:12.855502 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.851187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34659 (local address 127.1.163.67:40181)
0925 15:25:12.851418 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.851427 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.851505 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.851662 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.851663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.851665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:12.851739 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.851740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.852628 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.852630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.854017 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.854020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.854975 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.854977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.855076 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.855082 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.855164 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.855214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":111,"thread_start_us":149,"threads_started":1}
I20260925 15:25:12.855839 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f94ff684fd47a7bb67bbe05be831f5" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:12.855845 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f94ff684fd47a7bb67bbe05be831f5" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63" is_pre_election: true
I20260925 15:25:12.856057 3714 raft_consensus.cc:2454] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:12.856035 3573 raft_consensus.cc:2454] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:12.856362 3788 leader_election.cc:302] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f48030e187f14316b8a901cb012eed43; no voters:
I20260925 15:25:12.856521 3931 raft_consensus.cc:2781] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:12.856624 3931 raft_consensus.cc:482] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:12.856666 3931 raft_consensus.cc:3037] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:12.857513 3931 raft_consensus.cc:504] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.857697 3931 leader_election.cc:288] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:12.857928 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f94ff684fd47a7bb67bbe05be831f5" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:12.857929 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f94ff684fd47a7bb67bbe05be831f5" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63"
I20260925 15:25:12.858035 3573 raft_consensus.cc:3037] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:12.858035 3714 raft_consensus.cc:3037] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:12.859005 3573 raft_consensus.cc:2454] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 1.
I20260925 15:25:12.859141 3714 raft_consensus.cc:2454] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 1.
I20260925 15:25:12.859238 3788 leader_election.cc:302] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f48030e187f14316b8a901cb012eed43; no voters:
I20260925 15:25:12.859386 3931 raft_consensus.cc:2781] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:12.859555 3931 raft_consensus.cc:686] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 1 LEADER]: Becoming Leader. State: Replica: f48030e187f14316b8a901cb012eed43, State: Running, Role: LEADER
I20260925 15:25:12.859687 3931 consensus_queue.cc:237] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } }
I20260925 15:25:12.860633 3417 catalog_manager.cc:5495] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48030e187f14316b8a901cb012eed43 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "f48030e187f14316b8a901cb012eed43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } health_report { overall_health: UNKNOWN } } }
W20260925 15:25:12.892162 3901 tablet.cc:2369] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:25:12 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:12.901517 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.893330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44617 (local address 127.0.0.1:58568)
0925 15:25:12.893563 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:12.893583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:12.893682 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:12.893902 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:12.893903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:12.894301 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:12.894365 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.894367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.895274 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.895276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.895888 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:12.895890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.896334 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.896335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:12.896396 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.897055 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:12.897061 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:12.898651 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:12.900341 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.900350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.900352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.900649 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.900862 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:12.900863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:12.900863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:12.900929 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:12.901186 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:12.901193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:12.901239 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:25:12.901314 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:25:12.901419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260925 15:25:12.901588 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:12.893366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.1.163.126:44617)
0925 15:25:12.893492 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:12.893495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:12.893659 (+ 164us) server_negotiation.cc:407] Connection header received
0925 15:25:12.893753 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:12.893754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:12.893766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:12.893875 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:12.894525 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.895167 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.895998 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:12.896199 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:12.896329 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:12.898745 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:12.898748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:12.898748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:12.898756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:12.900240 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.900725 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.900726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.900726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.900779 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:12.901018 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:12.901019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:12.901019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:12.901128 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:12.901181 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:12.901345 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:25:12.901465 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":58}
I20260925 15:25:12.903898 3417 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:58568:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:25:12.904112 3417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:12.905815 3554 tablet_service.cc:1467] Processing CreateTablet for tablet 0cef6dc63eda444f92326171d7a7f00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7fa0fc2c0a40c49a135327f6a2d12e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:12.905833 3834 tablet_service.cc:1467] Processing CreateTablet for tablet 0cef6dc63eda444f92326171d7a7f00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7fa0fc2c0a40c49a135327f6a2d12e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:12.905820 3694 tablet_service.cc:1467] Processing CreateTablet for tablet 0cef6dc63eda444f92326171d7a7f00b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7fa0fc2c0a40c49a135327f6a2d12e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:12.906001 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cef6dc63eda444f92326171d7a7f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.906002 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cef6dc63eda444f92326171d7a7f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.906004 3694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cef6dc63eda444f92326171d7a7f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:12.907440 3925 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: Bootstrap starting.
I20260925 15:25:12.907539 3923 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: Bootstrap starting.
I20260925 15:25:12.907539 3924 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: Bootstrap starting.
I20260925 15:25:12.907820 3925 tablet_bootstrap.cc:654] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.907991 3924 tablet_bootstrap.cc:654] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.908272 3923 tablet_bootstrap.cc:654] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:12.908417 3925 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: No bootstrap required, opened a new log
I20260925 15:25:12.908466 3925 ts_tablet_manager.cc:1397] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.908679 3925 raft_consensus.cc:348] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.908764 3925 raft_consensus.cc:374] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.908795 3925 raft_consensus.cc:729] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e291f1fb77455194b8b00873c9a9a5, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.908804 3924 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: No bootstrap required, opened a new log
I20260925 15:25:12.908845 3924 ts_tablet_manager.cc:1397] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:12.908845 3925 consensus_queue.cc:260] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.908958 3925 ts_tablet_manager.cc:1428] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:12.909008 3923 tablet_bootstrap.cc:492] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: No bootstrap required, opened a new log
I20260925 15:25:12.908984 3930 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:12.909044 3923 ts_tablet_manager.cc:1397] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:12.908993 3924 raft_consensus.cc:348] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.909049 3930 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:12.909070 3924 raft_consensus.cc:374] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.909096 3924 raft_consensus.cc:729] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48030e187f14316b8a901cb012eed43, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.909147 3924 consensus_queue.cc:260] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.909224 3924 ts_tablet_manager.cc:1428] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:12.909184 3923 raft_consensus.cc:348] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.909263 3923 raft_consensus.cc:374] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:12.909287 3923 raft_consensus.cc:729] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Initialized, Role: FOLLOWER
I20260925 15:25:12.909336 3923 consensus_queue.cc:260] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:12.909344 3931 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:12.909405 3931 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:12.909431 3923 ts_tablet_manager.cc:1428] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:12.909510 3929 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:12.909614 3929 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:13.008980 3929 raft_consensus.cc:482] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:13.009114 3929 raft_consensus.cc:504] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.009503 3929 leader_election.cc:288] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:13.012701 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.009929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:55701 (local address 127.1.163.65:33263)
0925 15:25:13.009982 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.009984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.009998 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:25:13.010093 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.010093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.010103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.010177 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:13.010385 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.010874 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.011947 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.012305 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.012392 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.012402 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.012581 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.012619 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:25:13.012706 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.009982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:49969 (local address 127.1.163.67:41187)
0925 15:25:13.010025 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.010027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.010102 (+ 75us) server_negotiation.cc:407] Connection header received
0925 15:25:13.010174 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.010175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.010185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.010261 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:13.010421 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.010887 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.011948 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.012319 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.012403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.012411 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.012595 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.012634 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:25:13.012748 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.009731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33263 (local address 127.1.163.66:55701)
0925 15:25:13.009904 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.009924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.010018 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.010183 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.010184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.010187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:13.010251 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.010253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.010982 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.010983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.011846 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.011848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.012391 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.012392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.012467 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.012471 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.012535 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.012602 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":36,"thread_start_us":76,"threads_started":1}
I20260925 15:25:13.012753 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.009920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:41187 (local address 127.1.163.66:49969)
0925 15:25:13.010029 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.010039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.010101 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.010267 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.010268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.010270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:13.010342 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.010348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.011031 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.011033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.011845 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.011847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.012414 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.012415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.012496 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.012500 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.012588 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.012633 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":4,"thread_start_us":37,"threads_started":1}
I20260925 15:25:13.013118 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cef6dc63eda444f92326171d7a7f00b" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:13.013146 3854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cef6dc63eda444f92326171d7a7f00b" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43" is_pre_election: true
I20260925 15:25:13.013230 3573 raft_consensus.cc:2454] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:13.013284 3854 raft_consensus.cc:2454] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:13.013459 3648 leader_election.cc:302] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48030e187f14316b8a901cb012eed43, f6991e6957d14551bf3519aea29e6e63; no voters:
I20260925 15:25:13.013622 3929 raft_consensus.cc:2781] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:13.013681 3929 raft_consensus.cc:482] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:13.013698 3929 raft_consensus.cc:3037] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.014418 3929 raft_consensus.cc:504] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.014555 3929 leader_election.cc:288] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:13.014762 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cef6dc63eda444f92326171d7a7f00b" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:13.014817 3854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cef6dc63eda444f92326171d7a7f00b" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43"
I20260925 15:25:13.014887 3854 raft_consensus.cc:3037] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.014870 3573 raft_consensus.cc:3037] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.015578 3854 raft_consensus.cc:2454] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 1.
I20260925 15:25:13.015692 3573 raft_consensus.cc:2454] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 1.
I20260925 15:25:13.015743 3648 leader_election.cc:302] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48030e187f14316b8a901cb012eed43, f6991e6957d14551bf3519aea29e6e63; no voters:
I20260925 15:25:13.015865 3929 raft_consensus.cc:2781] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:13.016052 3929 raft_consensus.cc:686] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [term 1 LEADER]: Becoming Leader. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Running, Role: LEADER
I20260925 15:25:13.016173 3929 consensus_queue.cc:237] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.016669 3946 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: TxnStatusTablet state changed. Reason: New leader f6991e6957d14551bf3519aea29e6e63. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.016772 3946 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:13.016970 3948 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:25:13.017122 3948 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:25:13.017241 3416 catalog_manager.cc:5495] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6991e6957d14551bf3519aea29e6e63 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:13.034844 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.031570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.1.163.66:34659)
0925 15:25:13.031635 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.031637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.031649 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:13.031763 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.031764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.031787 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.031864 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.032070 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.032913 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.033819 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.034134 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.034210 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.034451 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.034614 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.034707 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.034725 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.034750 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:13.034842 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.031398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34659 (local address 127.0.0.1:47078)
0925 15:25:13.031482 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.031555 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.031612 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.031937 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.031939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.031940 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:13.031990 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.031992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.033001 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.033002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.033713 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.033716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.034237 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.034238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.034300 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.034332 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.034683 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.034690 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.034720 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.034773 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:25:13.037214 3854 raft_consensus.cc:1261] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.037242 3573 raft_consensus.cc:1261] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.037637 3929 consensus_queue.cc:1035] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.037819 3946 consensus_queue.cc:1035] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.039078 3931 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: TxnStatusTablet state changed. Reason: New leader f6991e6957d14551bf3519aea29e6e63. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.039083 3930 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: TxnStatusTablet state changed. Reason: New leader f6991e6957d14551bf3519aea29e6e63. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.039178 3931 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:13.039179 3930 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:13.040205 3929 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.040297 3929 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:13.040582 3951 mvcc.cc:205] Tried to move back new op lower bound from 7333273243797114880 to 7333273243714924544. Current Snapshot: MvccSnapshot[applied={T|T < 7333273243797114880}]
I20260925 15:25:13.040884 3946 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.040961 3946 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f6991e6957d14551bf3519aea29e6e63: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:13.041137 3930 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.041230 3930 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f1e291f1fb77455194b8b00873c9a9a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:13.041391 3931 tablet_replica.cc:439] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } } }
I20260925 15:25:13.041482 3931 tablet_replica.cc:442] T 0cef6dc63eda444f92326171d7a7f00b P f48030e187f14316b8a901cb012eed43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:13.042519 3952 mvcc.cc:205] Tried to move back new op lower bound from 7333273243797114880 to 7333273243714924544. Current Snapshot: MvccSnapshot[applied={T|T < 7333273243797114880}]
I20260925 15:25:13.046068 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.042951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.1.163.67:41187)
0925 15:25:13.043050 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.043052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.043061 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:13.043089 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.043089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.043106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.043182 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.043328 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.044148 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.045203 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.045541 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.045628 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.045744 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.045868 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.045947 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.045972 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.045993 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:25:13.051669 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.049030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:36805 (local address 127.1.163.66:34659)
0925 15:25:13.049091 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.049093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.049105 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:13.049135 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.049136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.049145 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.049199 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.049341 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.049817 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.050917 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.051130 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.051197 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.051423 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.051454 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.051518 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.051586 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.051605 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:13.051707 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.048941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34659 (local address 127.1.163.67:36805)
0925 15:25:13.049019 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.049030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.049068 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.049217 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.049218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.049219 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:13.049279 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.049280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.049891 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.049892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.050809 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.050811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.051211 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.051212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.051285 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.051301 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.051527 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.051529 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.051583 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.051628 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:25:13.054349 3573 raft_consensus.cc:1261] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Refusing update from remote peer f48030e187f14316b8a901cb012eed43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.054454 3714 raft_consensus.cc:1261] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Refusing update from remote peer f48030e187f14316b8a901cb012eed43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.054629 3933 consensus_queue.cc:1035] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.054749 3931 consensus_queue.cc:1035] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.065416 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.062096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40297 (local address 127.1.163.67:41187)
0925 15:25:13.062136 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.062138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.062147 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:13.062202 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.062203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.062212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.062274 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.062372 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.063118 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.064562 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.064872 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.064944 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.065151 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.065184 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.065236 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.065320 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.065342 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:13.065456 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.061911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:41187 (local address 127.1.163.66:40297)
0925 15:25:13.062079 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.062092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.062144 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.062276 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.062276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.062277 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:13.062333 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.062334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.063312 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.063314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.064452 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.064455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.064974 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.064975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.065042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.065061 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.065251 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.065253 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.065315 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.065356 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260925 15:25:13.077127 3416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:13.077271 3416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:13.078169 3416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Sending DeleteTablet for 3 replicas of tablet 71f94ff684fd47a7bb67bbe05be831f5
I20260925 15:25:13.078495 3694 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:13 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260925 15:25:13.078601 3554 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:13 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:54450
I20260925 15:25:13.078900 3834 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f94ff684fd47a7bb67bbe05be831f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:13 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:57902
I20260925 15:25:13.079087 3416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:25:13.079176 3971 tablet_replica.cc:330] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: stopping tablet replica
W20260925 15:25:13.079258 3416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:13.079305 3971 raft_consensus.cc:2229] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:13.079365 3971 raft_consensus.cc:2258] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:13.079492 3973 tablet_replica.cc:330] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: stopping tablet replica
I20260925 15:25:13.079571 3973 raft_consensus.cc:2229] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:13.080693 3694 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4e3defe0d449f287a1668428492028 (DEFAULT_TABLE table=test-table [id=690244d6071a4502a4fb947d8b3f3d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:13.080713 3834 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4e3defe0d449f287a1668428492028 (DEFAULT_TABLE table=test-table [id=690244d6071a4502a4fb947d8b3f3d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:13.080844 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4e3defe0d449f287a1668428492028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:13.080843 3694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4e3defe0d449f287a1668428492028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:13.082355 3923 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Bootstrap starting.
I20260925 15:25:13.082369 3924 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Bootstrap starting.
I20260925 15:25:13.082794 3923 tablet_bootstrap.cc:654] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:13.082832 3924 tablet_bootstrap.cc:654] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:13.083364 3554 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4e3defe0d449f287a1668428492028 (DEFAULT_TABLE table=test-table [id=690244d6071a4502a4fb947d8b3f3d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:13.083511 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4e3defe0d449f287a1668428492028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:13.083779 3973 raft_consensus.cc:2258] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:13.084323 3971 ts_tablet_manager.cc:1905] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:13.084868 3925 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Bootstrap starting.
I20260925 15:25:13.084903 3973 ts_tablet_manager.cc:1905] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:13.085292 3925 tablet_bootstrap.cc:654] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:13.085572 3924 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: No bootstrap required, opened a new log
I20260925 15:25:13.085640 3924 ts_tablet_manager.cc:1397] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:25:13.085812 3924 raft_consensus.cc:348] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.085891 3924 raft_consensus.cc:374] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:13.085913 3924 raft_consensus.cc:729] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48030e187f14316b8a901cb012eed43, State: Initialized, Role: FOLLOWER
I20260925 15:25:13.085953 3924 consensus_queue.cc:260] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.086047 3924 ts_tablet_manager.cc:1428] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:13.087723 3973 ts_tablet_manager.cc:1918] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:13.087865 3973 log.cc:1196] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/71f94ff684fd47a7bb67bbe05be831f5
I20260925 15:25:13.088229 3973 ts_tablet_manager.cc:1939] T 71f94ff684fd47a7bb67bbe05be831f5 P f48030e187f14316b8a901cb012eed43: Deleting consensus metadata
I20260925 15:25:13.088774 3404 catalog_manager.cc:4841] TS f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187): tablet 71f94ff684fd47a7bb67bbe05be831f5 (table test-table [id=0386e9300e724f25b89feb368a9563de]) successfully deleted
I20260925 15:25:13.089092 3971 ts_tablet_manager.cc:1918] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:13.089181 3971 log.cc:1196] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/71f94ff684fd47a7bb67bbe05be831f5
I20260925 15:25:13.089655 3971 ts_tablet_manager.cc:1939] T 71f94ff684fd47a7bb67bbe05be831f5 P f6991e6957d14551bf3519aea29e6e63: Deleting consensus metadata
I20260925 15:25:13.090181 3401 catalog_manager.cc:4841] TS f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659): tablet 71f94ff684fd47a7bb67bbe05be831f5 (table test-table [id=0386e9300e724f25b89feb368a9563de]) successfully deleted
I20260925 15:25:13.092671 3972 tablet_replica.cc:330] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: stopping tablet replica
I20260925 15:25:13.092891 3972 raft_consensus.cc:2229] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:13.092994 3972 raft_consensus.cc:2258] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:13.093017 3925 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: No bootstrap required, opened a new log
I20260925 15:25:13.093101 3925 ts_tablet_manager.cc:1397] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260925 15:25:13.093277 3925 raft_consensus.cc:348] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.093367 3925 raft_consensus.cc:374] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:13.093391 3925 raft_consensus.cc:729] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e291f1fb77455194b8b00873c9a9a5, State: Initialized, Role: FOLLOWER
I20260925 15:25:13.093451 3925 consensus_queue.cc:260] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.093526 3923 tablet_bootstrap.cc:492] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: No bootstrap required, opened a new log
I20260925 15:25:13.093565 3925 ts_tablet_manager.cc:1428] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:13.093582 3923 ts_tablet_manager.cc:1397] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260925 15:25:13.093725 3972 ts_tablet_manager.cc:1905] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:13.093703 3923 raft_consensus.cc:348] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.093765 3923 raft_consensus.cc:374] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:13.093782 3923 raft_consensus.cc:729] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Initialized, Role: FOLLOWER
I20260925 15:25:13.093825 3923 consensus_queue.cc:260] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.093909 3923 ts_tablet_manager.cc:1428] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:13.095780 3972 ts_tablet_manager.cc:1918] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:13.095863 3972 log.cc:1196] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/71f94ff684fd47a7bb67bbe05be831f5
I20260925 15:25:13.096127 3972 ts_tablet_manager.cc:1939] T 71f94ff684fd47a7bb67bbe05be831f5 P f1e291f1fb77455194b8b00873c9a9a5: Deleting consensus metadata
I20260925 15:25:13.096683 3404 catalog_manager.cc:4841] TS f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263): tablet 71f94ff684fd47a7bb67bbe05be831f5 (table test-table [id=0386e9300e724f25b89feb368a9563de]) successfully deleted
I20260925 15:25:13.212527 3969 raft_consensus.cc:482] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:13.212786 3931 raft_consensus.cc:482] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:13.212734 3969 raft_consensus.cc:504] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.212859 3931 raft_consensus.cc:504] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.212944 3969 leader_election.cc:288] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:13.212996 3931 leader_election.cc:288] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:13.213230 3574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:13.213313 3854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43" is_pre_election: true
I20260925 15:25:13.213382 3574 raft_consensus.cc:2454] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:13.213323 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63" is_pre_election: true
I20260925 15:25:13.213423 3714 raft_consensus.cc:2454] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:13.213423 3854 raft_consensus.cc:2454] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:13.213253 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:13.213564 3573 raft_consensus.cc:2454] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:13.213569 3788 leader_election.cc:302] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f48030e187f14316b8a901cb012eed43; no voters:
I20260925 15:25:13.213586 3648 leader_election.cc:302] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48030e187f14316b8a901cb012eed43, f6991e6957d14551bf3519aea29e6e63; no voters:
I20260925 15:25:13.213712 3931 raft_consensus.cc:2781] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:13.213732 3969 raft_consensus.cc:2781] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:13.213765 3969 raft_consensus.cc:482] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:13.213768 3931 raft_consensus.cc:482] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:13.213785 3969 raft_consensus.cc:3037] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.213797 3931 raft_consensus.cc:3037] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.214473 3969 raft_consensus.cc:504] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.214475 3931 raft_consensus.cc:504] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.214615 3969 leader_election.cc:288] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:13.214624 3931 leader_election.cc:288] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:13.214768 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:13.214841 3573 raft_consensus.cc:3037] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:13.214829 3854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43"
I20260925 15:25:13.214859 3574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:13.214905 3854 raft_consensus.cc:2379] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6991e6957d14551bf3519aea29e6e63 in current term 1: Already voted for candidate f48030e187f14316b8a901cb012eed43 in this term.
I20260925 15:25:13.215019 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4e3defe0d449f287a1668428492028" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63"
I20260925 15:25:13.215116 3714 raft_consensus.cc:2379] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f48030e187f14316b8a901cb012eed43 in current term 1: Already voted for candidate f6991e6957d14551bf3519aea29e6e63 in this term.
I20260925 15:25:13.215314 3785 leader_election.cc:302] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f48030e187f14316b8a901cb012eed43; no voters: f1e291f1fb77455194b8b00873c9a9a5, f6991e6957d14551bf3519aea29e6e63
I20260925 15:25:13.215437 3931 raft_consensus.cc:2726] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:25:13.215667 3573 raft_consensus.cc:2454] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 1.
I20260925 15:25:13.215858 3648 leader_election.cc:302] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f6991e6957d14551bf3519aea29e6e63; no voters: f48030e187f14316b8a901cb012eed43
I20260925 15:25:13.215962 3969 raft_consensus.cc:2781] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:13.216025 3969 raft_consensus.cc:686] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 LEADER]: Becoming Leader. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Running, Role: LEADER
I20260925 15:25:13.216078 3969 consensus_queue.cc:237] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:13.216722 3416 catalog_manager.cc:5495] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6991e6957d14551bf3519aea29e6e63 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:13.236919 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.234287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.1.163.66:34659)
0925 15:25:13.234363 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.234365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.234375 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:13.234464 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.234464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.234475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.234531 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.234659 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.235165 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.236163 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.236395 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.236481 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.236639 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.236680 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.236740 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.236794 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.236818 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:25:13.241716 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.239355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:48149 (local address 127.1.163.66:34659)
0925 15:25:13.239403 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:13.239405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:13.239413 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:13.239434 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:13.239435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:13.239444 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:13.239495 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:13.239649 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.240132 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.241026 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.241197 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.241286 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.241441 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.241477 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.241537 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:13.241586 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:25:13.241611 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":32}
I20260925 15:25:13.241716 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:13.239262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34659 (local address 127.1.163.66:48149)
0925 15:25:13.239322 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:13.239333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:13.239385 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:13.239536 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:13.239537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:13.239538 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:13.239575 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.239576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.240227 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.240228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.240907 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:13.240909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:13.241299 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:13.241300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:13.241348 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:13.241353 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:13.241536 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:13.241537 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:25:13.241586 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:25:13.241639 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":2}
I20260925 15:25:13.244019 3854 raft_consensus.cc:1261] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.244027 3573 raft_consensus.cc:1261] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:13.244243 3929 consensus_queue.cc:1035] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.244355 3946 consensus_queue.cc:1035] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:13.245740 3962 mvcc.cc:205] Tried to move back new op lower bound from 7333273244645658624 to 7333273244533420032. Current Snapshot: MvccSnapshot[applied={T|T < 7333273244645658624}]
W20260925 15:25:13.249359 3951 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:25:14.773157 3416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:14.773324 3416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:14.774235 3416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Sending DeleteTablet for 3 replicas of tablet 2d4e3defe0d449f287a1668428492028
I20260925 15:25:14.774552 3554 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4e3defe0d449f287a1668428492028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:14 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:54450
I20260925 15:25:14.774551 3834 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4e3defe0d449f287a1668428492028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:14 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:57902
I20260925 15:25:14.774757 4001 tablet_replica.cc:330] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: stopping tablet replica
I20260925 15:25:14.774855 4002 tablet_replica.cc:330] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: stopping tablet replica
I20260925 15:25:14.774861 4001 raft_consensus.cc:2229] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:14.774910 4001 raft_consensus.cc:2258] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:14.774916 4002 raft_consensus.cc:2229] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:14.775142 4002 raft_consensus.cc:2258] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:14.775184 4001 ts_tablet_manager.cc:1905] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:14.775396 4002 ts_tablet_manager.cc:1905] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:14.777025 4001 ts_tablet_manager.cc:1918] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:25:14.777150 4001 log.cc:1196] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/2d4e3defe0d449f287a1668428492028
I20260925 15:25:14.777174 4002 ts_tablet_manager.cc:1918] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:25:14.777244 4002 log.cc:1196] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/2d4e3defe0d449f287a1668428492028
I20260925 15:25:14.777251 3694 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4e3defe0d449f287a1668428492028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:14 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260925 15:25:14.777480 4001 ts_tablet_manager.cc:1939] T 2d4e3defe0d449f287a1668428492028 P f1e291f1fb77455194b8b00873c9a9a5: Deleting consensus metadata
I20260925 15:25:14.777495 4002 ts_tablet_manager.cc:1939] T 2d4e3defe0d449f287a1668428492028 P f48030e187f14316b8a901cb012eed43: Deleting consensus metadata
I20260925 15:25:14.777889 3404 catalog_manager.cc:4841] TS f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263): tablet 2d4e3defe0d449f287a1668428492028 (table test-table [id=690244d6071a4502a4fb947d8b3f3d79]) successfully deleted
I20260925 15:25:14.777988 3404 catalog_manager.cc:4841] TS f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187): tablet 2d4e3defe0d449f287a1668428492028 (table test-table [id=690244d6071a4502a4fb947d8b3f3d79]) successfully deleted
I20260925 15:25:14.778266 4004 tablet_replica.cc:330] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: stopping tablet replica
I20260925 15:25:14.778328 4004 raft_consensus.cc:2229] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:14.778425 4004 raft_consensus.cc:2258] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:14.778652 4004 ts_tablet_manager.cc:1905] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:14.780169 4004 ts_tablet_manager.cc:1918] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:25:14.780239 4004 log.cc:1196] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/2d4e3defe0d449f287a1668428492028
I20260925 15:25:14.780500 4004 ts_tablet_manager.cc:1939] T 2d4e3defe0d449f287a1668428492028 P f6991e6957d14551bf3519aea29e6e63: Deleting consensus metadata
I20260925 15:25:14.780929 3401 catalog_manager.cc:4841] TS f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659): tablet 2d4e3defe0d449f287a1668428492028 (table test-table [id=690244d6071a4502a4fb947d8b3f3d79]) successfully deleted
Sep 25 15:25:14 dist-test-slave-hvv5 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:25:14 dist-test-slave-hvv5 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:14.800310 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:14.792227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.1.163.126:44617)
0925 15:25:14.792503 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:14.792505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:14.792515 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:14.792582 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:14.792582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:14.792595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:14.792676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:14.793298 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:14.793870 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:14.794726 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:14.795002 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:14.795076 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:14.797636 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:14.797637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:14.797637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:14.797644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:14.799146 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:14.799477 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:14.799478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:14.799479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:14.799538 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:14.799729 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:14.799730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:14.799730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:14.799875 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:14.799961 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:14.800143 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 15:25:14.800222 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":215,"threads_started":1}
I20260925 15:25:14.801951 3416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:14.802100 3416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:14.803525 3554 tablet_service.cc:1467] Processing CreateTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (DEFAULT_TABLE table=test-table [id=cee956335a884a6a952706b86a1f8617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:14.803525 3834 tablet_service.cc:1467] Processing CreateTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (DEFAULT_TABLE table=test-table [id=cee956335a884a6a952706b86a1f8617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:14.803593 3694 tablet_service.cc:1467] Processing CreateTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (DEFAULT_TABLE table=test-table [id=cee956335a884a6a952706b86a1f8617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:14.803668 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffabea1cb8a4b4e9e40da9b71ee3c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:14.803668 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffabea1cb8a4b4e9e40da9b71ee3c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:14.803738 3694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffabea1cb8a4b4e9e40da9b71ee3c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:14.805126 4014 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Bootstrap starting.
I20260925 15:25:14.805126 4013 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Bootstrap starting.
I20260925 15:25:14.805275 4012 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Bootstrap starting.
I20260925 15:25:14.805549 4013 tablet_bootstrap.cc:654] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:14.805619 4014 tablet_bootstrap.cc:654] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:14.805629 4012 tablet_bootstrap.cc:654] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:14.806205 4013 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: No bootstrap required, opened a new log
I20260925 15:25:14.806249 4014 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: No bootstrap required, opened a new log
I20260925 15:25:14.806258 4012 tablet_bootstrap.cc:492] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: No bootstrap required, opened a new log
I20260925 15:25:14.806258 4013 ts_tablet_manager.cc:1397] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:14.806296 4014 ts_tablet_manager.cc:1397] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:14.806300 4012 ts_tablet_manager.cc:1397] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:14.806378 4013 raft_consensus.cc:348] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806434 4013 raft_consensus.cc:374] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:14.806447 4013 raft_consensus.cc:729] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48030e187f14316b8a901cb012eed43, State: Initialized, Role: FOLLOWER
I20260925 15:25:14.806411 4014 raft_consensus.cc:348] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806403 4012 raft_consensus.cc:348] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806473 4014 raft_consensus.cc:374] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:14.806473 4012 raft_consensus.cc:374] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:14.806495 4014 raft_consensus.cc:729] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e291f1fb77455194b8b00873c9a9a5, State: Initialized, Role: FOLLOWER
I20260925 15:25:14.806495 4012 raft_consensus.cc:729] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Initialized, Role: FOLLOWER
I20260925 15:25:14.806516 4013 consensus_queue.cc:260] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806567 4012 consensus_queue.cc:260] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806584 4014 consensus_queue.cc:260] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:14.806674 4012 ts_tablet_manager.cc:1428] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:14.806699 4013 ts_tablet_manager.cc:1428] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:14.807444 4014 ts_tablet_manager.cc:1428] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:15.004570 3929 raft_consensus.cc:482] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:15.004686 3929 raft_consensus.cc:504] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.005175 3929 leader_election.cc:288] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:15.005508 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffabea1cb8a4b4e9e40da9b71ee3c74" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:15.005596 3573 raft_consensus.cc:2454] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:15.005573 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffabea1cb8a4b4e9e40da9b71ee3c74" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43" is_pre_election: true
I20260925 15:25:15.005715 3853 raft_consensus.cc:2454] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 0.
I20260925 15:25:15.005880 3648 leader_election.cc:302] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48030e187f14316b8a901cb012eed43, f6991e6957d14551bf3519aea29e6e63; no voters:
I20260925 15:25:15.006013 3929 raft_consensus.cc:2781] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:15.006073 3929 raft_consensus.cc:482] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:15.006094 3929 raft_consensus.cc:3037] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.006907 3929 raft_consensus.cc:504] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.007146 3929 leader_election.cc:288] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187)
I20260925 15:25:15.007382 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffabea1cb8a4b4e9e40da9b71ee3c74" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:15.007385 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffabea1cb8a4b4e9e40da9b71ee3c74" candidate_uuid: "f6991e6957d14551bf3519aea29e6e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48030e187f14316b8a901cb012eed43"
I20260925 15:25:15.007467 3573 raft_consensus.cc:3037] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.007489 3853 raft_consensus.cc:3037] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.008147 3853 raft_consensus.cc:2454] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 1.
I20260925 15:25:15.008315 3573 raft_consensus.cc:2454] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6991e6957d14551bf3519aea29e6e63 in term 1.
I20260925 15:25:15.008419 3648 leader_election.cc:302] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f48030e187f14316b8a901cb012eed43, f6991e6957d14551bf3519aea29e6e63; no voters:
I20260925 15:25:15.008602 3929 raft_consensus.cc:2781] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:15.008687 3929 raft_consensus.cc:686] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 1 LEADER]: Becoming Leader. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Running, Role: LEADER
I20260925 15:25:15.008736 3929 consensus_queue.cc:237] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.009591 3416 catalog_manager.cc:5495] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6991e6957d14551bf3519aea29e6e63 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "f6991e6957d14551bf3519aea29e6e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:15.039237 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:15.036670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.1.163.66:34659)
0925 15:25:15.036816 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:15.036818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:15.036824 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:25:15.036846 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:15.036846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:15.036864 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:15.037005 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:15.037173 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:15.037680 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:15.038484 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:15.038703 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:15.038778 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:15.038950 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:15.038989 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:15.039058 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:15.039115 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:25:15.039139 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260925 15:25:15.041791 3573 raft_consensus.cc:1261] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:15.041771 3853 raft_consensus.cc:1261] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Refusing update from remote peer f6991e6957d14551bf3519aea29e6e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:15.042017 3929 consensus_queue.cc:1035] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:15.042137 3993 consensus_queue.cc:1035] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:15.052758 3416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:15.052887 3416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:15.053587 3416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Sending DeleteTablet for 3 replicas of tablet bffabea1cb8a4b4e9e40da9b71ee3c74
I20260925 15:25:15.053860 3554 tablet_service.cc:1514] Processing DeleteTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:15 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:54450
I20260925 15:25:15.053877 3694 tablet_service.cc:1514] Processing DeleteTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:15 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260925 15:25:15.053967 4001 tablet_replica.cc:330] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: stopping tablet replica
I20260925 15:25:15.053970 3834 tablet_service.cc:1514] Processing DeleteTablet for tablet bffabea1cb8a4b4e9e40da9b71ee3c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:15 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:57902
I20260925 15:25:15.054020 4001 raft_consensus.cc:2229] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:15.054060 4001 raft_consensus.cc:2258] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:15.054068 4002 tablet_replica.cc:330] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: stopping tablet replica
I20260925 15:25:15.054147 4002 raft_consensus.cc:2229] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:15.054165 4004 tablet_replica.cc:330] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: stopping tablet replica
I20260925 15:25:15.054201 4002 raft_consensus.cc:2258] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:15.054231 4004 raft_consensus.cc:2229] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:15.054292 4004 raft_consensus.cc:2258] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:15.054544 4002 ts_tablet_manager.cc:1905] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:15.054714 4001 ts_tablet_manager.cc:1905] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:15.054697 3416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:15.054837 3416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:15.056103 3694 tablet_service.cc:1467] Processing CreateTablet for tablet aeb5965ba479423dbed5618212806c37 (DEFAULT_TABLE table=test-table [id=b94fac4f0d2c446e96a2224b14fef578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:15.056221 3694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb5965ba479423dbed5618212806c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:15.056318 3554 tablet_service.cc:1467] Processing CreateTablet for tablet aeb5965ba479423dbed5618212806c37 (DEFAULT_TABLE table=test-table [id=b94fac4f0d2c446e96a2224b14fef578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:15.056635 3834 tablet_service.cc:1467] Processing CreateTablet for tablet aeb5965ba479423dbed5618212806c37 (DEFAULT_TABLE table=test-table [id=b94fac4f0d2c446e96a2224b14fef578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:15.056749 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb5965ba479423dbed5618212806c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:15.056741 4001 ts_tablet_manager.cc:1918] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:15.056793 4001 log.cc:1196] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/bffabea1cb8a4b4e9e40da9b71ee3c74
I20260925 15:25:15.057103 4001 ts_tablet_manager.cc:1939] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f1e291f1fb77455194b8b00873c9a9a5: Deleting consensus metadata
I20260925 15:25:15.057356 4002 ts_tablet_manager.cc:1918] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:15.057411 4002 log.cc:1196] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/bffabea1cb8a4b4e9e40da9b71ee3c74
I20260925 15:25:15.057621 4002 ts_tablet_manager.cc:1939] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f48030e187f14316b8a901cb012eed43: Deleting consensus metadata
I20260925 15:25:15.057960 4012 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Bootstrap starting.
I20260925 15:25:15.058055 4013 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Bootstrap starting.
I20260925 15:25:15.058080 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb5965ba479423dbed5618212806c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:15.058082 4004 ts_tablet_manager.cc:1905] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:15.058342 4012 tablet_bootstrap.cc:654] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:15.058360 4013 tablet_bootstrap.cc:654] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:15.059199 4014 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5: Bootstrap starting.
I20260925 15:25:15.059417 3404 catalog_manager.cc:4841] TS f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263): tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (table test-table [id=cee956335a884a6a952706b86a1f8617]) successfully deleted
I20260925 15:25:15.059492 3404 catalog_manager.cc:4841] TS f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187): tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (table test-table [id=cee956335a884a6a952706b86a1f8617]) successfully deleted
I20260925 15:25:15.059667 4004 ts_tablet_manager.cc:1918] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:15.059716 4004 log.cc:1196] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/bffabea1cb8a4b4e9e40da9b71ee3c74
I20260925 15:25:15.059773 4013 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: No bootstrap required, opened a new log
I20260925 15:25:15.059819 4013 ts_tablet_manager.cc:1397] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:15.059949 4004 ts_tablet_manager.cc:1939] T bffabea1cb8a4b4e9e40da9b71ee3c74 P f6991e6957d14551bf3519aea29e6e63: Deleting consensus metadata
I20260925 15:25:15.059989 4014 tablet_bootstrap.cc:654] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:15.059950 4013 raft_consensus.cc:348] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.060042 4013 raft_consensus.cc:374] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:15.060068 4013 raft_consensus.cc:729] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48030e187f14316b8a901cb012eed43, State: Initialized, Role: FOLLOWER
I20260925 15:25:15.060111 4012 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: No bootstrap required, opened a new log
I20260925 15:25:15.060110 4013 consensus_queue.cc:260] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.060159 4012 ts_tablet_manager.cc:1397] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:15.060199 4013 ts_tablet_manager.cc:1428] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:15.060276 4012 raft_consensus.cc:348] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.060350 4012 raft_consensus.cc:374] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:15.060376 4012 raft_consensus.cc:729] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6991e6957d14551bf3519aea29e6e63, State: Initialized, Role: FOLLOWER
I20260925 15:25:15.060410 4012 consensus_queue.cc:260] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.060503 4012 ts_tablet_manager.cc:1428] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:15.060663 3401 catalog_manager.cc:4841] TS f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659): tablet bffabea1cb8a4b4e9e40da9b71ee3c74 (table test-table [id=cee956335a884a6a952706b86a1f8617]) successfully deleted
I20260925 15:25:15.061097 4014 tablet_bootstrap.cc:492] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5: No bootstrap required, opened a new log
I20260925 15:25:15.061152 4014 ts_tablet_manager.cc:1397] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:15.061301 4014 raft_consensus.cc:348] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.061439 4014 raft_consensus.cc:374] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:15.061465 4014 raft_consensus.cc:729] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e291f1fb77455194b8b00873c9a9a5, State: Initialized, Role: FOLLOWER
I20260925 15:25:15.061508 4014 consensus_queue.cc:260] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.061619 4014 ts_tablet_manager.cc:1428] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:15.327493 4018 raft_consensus.cc:482] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:15.327603 4018 raft_consensus.cc:504] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.327772 4018 leader_election.cc:288] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:15.328027 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb5965ba479423dbed5618212806c37" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5" is_pre_election: true
I20260925 15:25:15.328107 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb5965ba479423dbed5618212806c37" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63" is_pre_election: true
I20260925 15:25:15.328159 3573 raft_consensus.cc:2454] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:15.328194 3714 raft_consensus.cc:2454] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 0.
I20260925 15:25:15.328338 3788 leader_election.cc:302] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f48030e187f14316b8a901cb012eed43; no voters:
I20260925 15:25:15.328446 4018 raft_consensus.cc:2781] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:15.328519 4018 raft_consensus.cc:482] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:15.328576 4018 raft_consensus.cc:3037] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.329325 4018 raft_consensus.cc:504] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.329478 4018 leader_election.cc:288] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Requested vote from peers f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263), f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659)
I20260925 15:25:15.329684 3573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb5965ba479423dbed5618212806c37" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e291f1fb77455194b8b00873c9a9a5"
I20260925 15:25:15.329774 3573 raft_consensus.cc:3037] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.329659 3714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb5965ba479423dbed5618212806c37" candidate_uuid: "f48030e187f14316b8a901cb012eed43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6991e6957d14551bf3519aea29e6e63"
I20260925 15:25:15.330000 3714 raft_consensus.cc:3037] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:15.330518 3573 raft_consensus.cc:2454] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 1.
I20260925 15:25:15.330696 3788 leader_election.cc:302] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e291f1fb77455194b8b00873c9a9a5, f48030e187f14316b8a901cb012eed43; no voters:
I20260925 15:25:15.330832 4018 raft_consensus.cc:2781] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:15.330828 3714 raft_consensus.cc:2454] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48030e187f14316b8a901cb012eed43 in term 1.
I20260925 15:25:15.330951 4018 raft_consensus.cc:686] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 1 LEADER]: Becoming Leader. State: Replica: f48030e187f14316b8a901cb012eed43, State: Running, Role: LEADER
I20260925 15:25:15.331012 4018 consensus_queue.cc:237] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } }
I20260925 15:25:15.331676 3417 catalog_manager.cc:5495] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 reported cstate change: term changed from 0 to 1, leader changed from <none> to f48030e187f14316b8a901cb012eed43 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "f48030e187f14316b8a901cb012eed43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48030e187f14316b8a901cb012eed43" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41187 } health_report { overall_health: HEALTHY } } }
I20260925 15:25:15.350131 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:15.347761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.1.163.67:41187)
0925 15:25:15.347933 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:15.347934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:15.347940 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:25:15.347958 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:15.347958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:15.347972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:15.348023 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:15.348159 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:15.348713 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:15.349486 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:15.349670 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:15.349722 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:15.349878 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:15.349904 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:15.349938 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:15.349996 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:25:15.350011 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":121,"threads_started":1}
I20260925 15:25:15.352479 3714 raft_consensus.cc:1261] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Refusing update from remote peer f48030e187f14316b8a901cb012eed43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:15.352499 3573 raft_consensus.cc:1261] T aeb5965ba479423dbed5618212806c37 P f1e291f1fb77455194b8b00873c9a9a5 [term 1 FOLLOWER]: Refusing update from remote peer f48030e187f14316b8a901cb012eed43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:15.352773 4029 consensus_queue.cc:1035] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6991e6957d14551bf3519aea29e6e63" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:15.353004 4018 consensus_queue.cc:1035] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e291f1fb77455194b8b00873c9a9a5" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:25:15.357117 3987 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:25:16.876077 3416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:16.876226 3416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:16.877254 3416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efb3c732a2c4edf88e218088bf01540: Sending DeleteTablet for 3 replicas of tablet aeb5965ba479423dbed5618212806c37
I20260925 15:25:16.877571 3554 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb5965ba479423dbed5618212806c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:16 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:54450
I20260925 15:25:16.877599 3834 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb5965ba479423dbed5618212806c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:16 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:57902
I20260925 15:25:16.877590 3693 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb5965ba479423dbed5618212806c37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:16 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:47074
I20260925 15:25:16.877820 4059 tablet_replica.cc:330] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: stopping tablet replica
I20260925 15:25:16.877893 4059 raft_consensus.cc:2229] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:16.877966 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3489
I20260925 15:25:16.877997 4059 raft_consensus.cc:2258] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:16.878767 4059 ts_tablet_manager.cc:1905] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:16.880846 4059 ts_tablet_manager.cc:1918] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:25:16.880905 4059 log.cc:1196] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/aeb5965ba479423dbed5618212806c37
I20260925 15:25:16.881141 4059 ts_tablet_manager.cc:1939] T aeb5965ba479423dbed5618212806c37 P f6991e6957d14551bf3519aea29e6e63: Deleting consensus metadata
I20260925 15:25:16.881796 3401 catalog_manager.cc:4841] TS f6991e6957d14551bf3519aea29e6e63 (127.1.163.66:34659): tablet aeb5965ba479423dbed5618212806c37 (table test-table [id=b94fac4f0d2c446e96a2224b14fef578]) successfully deleted
I20260925 15:25:16.884691 4057 tablet_replica.cc:330] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: stopping tablet replica
I20260925 15:25:16.884841 4057 raft_consensus.cc:2229] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:16.884945 4057 raft_consensus.cc:2258] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:25:16.885018 3404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:33263 (error 108)
I20260925 15:25:16.885223 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3629
I20260925 15:25:16.885288 4057 ts_tablet_manager.cc:1905] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:16.886346 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:16.886101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33263 (local address 127.0.0.1:54454)
0925 15:25:16.886258 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:16.886304 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260925 15:25:16.886432 3404 catalog_manager.cc:4584] TS f1e291f1fb77455194b8b00873c9a9a5 (127.1.163.65:33263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aeb5965ba479423dbed5618212806c37: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33263: connect: Connection refused (error 111)
I20260925 15:25:16.887785 4057 ts_tablet_manager.cc:1918] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:25:16.887849 4057 log.cc:1196] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/aeb5965ba479423dbed5618212806c37
I20260925 15:25:16.888082 4057 ts_tablet_manager.cc:1939] T aeb5965ba479423dbed5618212806c37 P f48030e187f14316b8a901cb012eed43: Deleting consensus metadata
I20260925 15:25:16.888516 3404 catalog_manager.cc:4841] TS f48030e187f14316b8a901cb012eed43 (127.1.163.67:41187): tablet aeb5965ba479423dbed5618212806c37 (table test-table [id=b94fac4f0d2c446e96a2224b14fef578]) successfully deleted
I20260925 15:25:16.892053 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3769
I20260925 15:25:16.899181 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 3385
2026-09-25T15:25:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6142 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:16 dist-test-slave-hvv5 krb5kdc[4064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:16 dist-test-slave-hvv5 krb5kdc[4064](info): set up 2 sockets
Sep 25 15:25:16 dist-test-slave-hvv5 krb5kdc[4064](info): commencing operation
krb5kdc: starting...
W20260925 15:25:17.956857 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:18.084014 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37553
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:18.174311 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:18.174573 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:18.174611 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:18.175808 4077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:18.175880 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:18.175906 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:18.175935 4077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:18.175964 4077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:18.177623 4077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:18.177927 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:18.178184 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:18.184425 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.184425 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.184459 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.186450 4077 server_base.cc:1034] running on GCE node
I20260925 15:25:18.186757 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:18.187150 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:18.188349 4077 hybrid_clock.cc:648] HybridClock initialized: now 1790349918188334 us; error 31 us; skew 500 ppm
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:18.198347 4077 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:18.198922 4077 webserver.cc:460] Webserver started at http://127.1.163.126:44873/ using document root <none> and password file <none>
I20260925 15:25:18.199208 4077 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:18.199282 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:18.199447 4077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:18.200680 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "0eb2ec8936334e57af80abf67d75c9cb"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "937e84de846c2b79a4f3f8f8026a2874"
server_key_iv: "62aa6c9b329089c3ad54dda03a87b9fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.201069 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "0eb2ec8936334e57af80abf67d75c9cb"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "937e84de846c2b79a4f3f8f8026a2874"
server_key_iv: "62aa6c9b329089c3ad54dda03a87b9fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.202955 4077 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:18.203778 4092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.204051 4077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:18.204174 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "0eb2ec8936334e57af80abf67d75c9cb"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "937e84de846c2b79a4f3f8f8026a2874"
server_key_iv: "62aa6c9b329089c3ad54dda03a87b9fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.204258 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:18.222859 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:18.225333 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:18.225530 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:18.230351 4077 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37553
I20260925 15:25:18.230427 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37553 every 8 connection(s)
I20260925 15:25:18.230998 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:18.231827 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:18.234293 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb: Bootstrap starting.
I20260925 15:25:18.235024 4145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:18.235415 4145 log.cc:824] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:18.236414 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb: No bootstrap required, opened a new log
I20260925 15:25:18.238140 4145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } }
I20260925 15:25:18.238296 4145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:18.238318 4145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb2ec8936334e57af80abf67d75c9cb, State: Initialized, Role: FOLLOWER
I20260925 15:25:18.238463 4145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } }
I20260925 15:25:18.238547 4145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:18.238570 4145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:18.238600 4145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:18.239739 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4077
I20260925 15:25:18.239840 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:18.239938 4145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } }
I20260925 15:25:18.240088 4145 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb2ec8936334e57af80abf67d75c9cb; no voters:
I20260925 15:25:18.240113 1677 external_mini_cluster.cc:1428] Setting key b954aef4ae4601538ed9d2d22840025e
I20260925 15:25:18.240316 4145 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:18.240631 4149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:18.240680 4145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:18.240962 4149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [term 1 LEADER]: Becoming Leader. State: Replica: 0eb2ec8936334e57af80abf67d75c9cb, State: Running, Role: LEADER
I20260925 15:25:18.241185 4149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } }
I20260925 15:25:18.241638 4151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb2ec8936334e57af80abf67d75c9cb. Latest consensus state: current_term: 1 leader_uuid: "0eb2ec8936334e57af80abf67d75c9cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } } }
I20260925 15:25:18.241729 4151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:18.241848 4148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb2ec8936334e57af80abf67d75c9cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb2ec8936334e57af80abf67d75c9cb" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37553 } } }
I20260925 15:25:18.241950 4148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:18.241999 4159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:18.242556 4159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:18.245302 4159 catalog_manager.cc:1347] Generated new cluster ID: 23d1c7404045432f9e29c8930832df21
I20260925 15:25:18.245359 4159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:18.253093 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.240974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.1.163.126:37553)
0925 15:25:18.241176 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:18.241186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:18.241223 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:25:18.241359 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:18.241368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:18.241429 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:18.242439 (+ 1010us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:18.242552 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.243789 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.244995 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.245609 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.245711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.248767 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:18.248771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:18.248772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:18.248784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:18.250855 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.251324 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.251327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.251328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.251449 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.251679 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.251680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.251680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.252070 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:18.252232 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:18.252654 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 15:25:18.252826 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260925 15:25:18.258352 4159 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:18.258991 4159 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:18.272142 4159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb2ec8936334e57af80abf67d75c9cb: Generated new TSK 0
I20260925 15:25:18.272408 4159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:18.307999 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:18.393474 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:18.393694 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:18.393723 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:18.394935 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:18.394981 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:18.395009 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:18.396513 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:18.396848 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:18.397111 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:18.402454 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.402953 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.402997 4171 server_base.cc:1034] running on GCE node
W20260925 15:25:18.403088 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.403317 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:18.403535 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:18.404678 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790349918404664 us; error 28 us; skew 500 ppm
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:18.407382 4171 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:18.407855 4171 webserver.cc:460] Webserver started at http://127.1.163.65:42161/ using document root <none> and password file <none>
I20260925 15:25:18.408167 4171 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:18.408236 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:18.408378 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:18.409602 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "12aa1cfc7eb04c5aa7839b6236990e2b"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "0d81e7030ccbe7616e67f7ce59f4b647"
server_key_iv: "91783c56fe2bc6792b34886012f9d19a"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.410022 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "12aa1cfc7eb04c5aa7839b6236990e2b"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "0d81e7030ccbe7616e67f7ce59f4b647"
server_key_iv: "91783c56fe2bc6792b34886012f9d19a"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.411432 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:18.412216 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.412381 4171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:18.412506 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "12aa1cfc7eb04c5aa7839b6236990e2b"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "0d81e7030ccbe7616e67f7ce59f4b647"
server_key_iv: "91783c56fe2bc6792b34886012f9d19a"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.412617 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:18.421685 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:18.423655 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:18.423820 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:18.424067 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:18.424389 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:18.424444 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.424490 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:18.424520 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.430562 4171 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:44239
I20260925 15:25:18.430603 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:44239 every 8 connection(s)
I20260925 15:25:18.430985 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:18.433425 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4171
I20260925 15:25:18.433512 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:18.433708 1677 external_mini_cluster.cc:1428] Setting key 27abcd2926e1cd4b444ddde473de9c6d
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:18.442698 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.431787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:41651 (local address 127.1.163.126:37553)
0925 15:25:18.431872 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:18.431875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:18.432170 (+ 295us) server_negotiation.cc:407] Connection header received
0925 15:25:18.432795 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:18.432797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:18.432812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:18.432877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:18.434003 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.434609 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.435629 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.435856 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.435945 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.438825 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:18.438827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:18.438827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:18.438872 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:18.440920 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.441389 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.441391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.441392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.441466 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.441879 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.441880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.441880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.442047 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:18.442120 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:18.442448 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:25:18.442548 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:25:18.442798 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.431736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37553 (local address 127.1.163.65:41651)
0925 15:25:18.431985 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:18.432034 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:18.432691 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:18.432938 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:18.432941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:18.433621 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:18.433900 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.433906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.434721 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.434722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.435536 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.435538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.435957 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.435959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.436097 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.436812 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:18.436821 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:18.438691 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:18.441024 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.441033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.441035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.441300 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.441683 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.441684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.441685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.441812 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.442127 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:18.442132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:18.442269 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:25:18.442412 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:25:18.442561 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:18.443333 4300 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37553
I20260925 15:25:18.443466 4300 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:18.443679 4300 heartbeater.cc:511] Master 127.1.163.126:37553 requested a full tablet report, sending...
I20260925 15:25:18.444195 4109 ts_manager.cc:194] Registered new tserver with Master: 12aa1cfc7eb04c5aa7839b6236990e2b (127.1.163.65:44239)
I20260925 15:25:18.445021 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:41651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:18.464035 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:18.548166 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:18.548437 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:18.548482 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:18.549779 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:18.549862 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:18.549911 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:18.551486 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:18.551781 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:18.552055 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:18.557477 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.557525 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.557494 4305 server_base.cc:1034] running on GCE node
W20260925 15:25:18.557525 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.557958 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:18.558168 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:18.559317 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790349918559305 us; error 28 us; skew 500 ppm
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:18.562029 4305 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:18.562543 4305 webserver.cc:460] Webserver started at http://127.1.163.66:36587/ using document root <none> and password file <none>
I20260925 15:25:18.562804 4305 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:18.562878 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:18.563030 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:18.564138 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "9bfdce6e42d34f93900507086b32ad88"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "14a2d7e637f1a3f9b05d46bd9e21d7d9"
server_key_iv: "0ed92a1252555bfd343df931219b8bef"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.564504 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "9bfdce6e42d34f93900507086b32ad88"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "14a2d7e637f1a3f9b05d46bd9e21d7d9"
server_key_iv: "0ed92a1252555bfd343df931219b8bef"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.565991 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:18.566788 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.566982 4305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:18.567085 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "9bfdce6e42d34f93900507086b32ad88"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "14a2d7e637f1a3f9b05d46bd9e21d7d9"
server_key_iv: "0ed92a1252555bfd343df931219b8bef"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.567167 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:18.593750 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:18.596278 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:18.596467 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:18.596769 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:18.597167 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:18.597225 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.597270 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:18.597317 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.603829 4305 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34327
I20260925 15:25:18.603869 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34327 every 8 connection(s)
I20260925 15:25:18.604220 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:18.609820 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4305
I20260925 15:25:18.609942 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:18.610159 1677 external_mini_cluster.cc:1428] Setting key 3e88fdcc1ddb89d39a776c97b40bfdf3
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:18.615355 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.605040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:59537 (local address 127.1.163.126:37553)
0925 15:25:18.605152 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:18.605154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:18.605340 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:25:18.606018 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:18.606019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:18.606032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:18.606075 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:18.607163 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.607703 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.608859 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.609064 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.609153 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.611883 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:18.611885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:18.611885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:18.611894 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:18.613521 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.614131 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.614131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.614132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.614186 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.614517 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.614517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.614517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.614624 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:18.614671 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:18.615138 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:25:18.615230 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:25:18.615720 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.604937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37553 (local address 127.1.163.66:59537)
0925 15:25:18.605192 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:18.605243 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:18.605850 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:18.606174 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:18.606177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:18.606860 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:18.607064 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.607069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.607801 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.607802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.608767 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.608770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.609162 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.609163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.609293 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.609850 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:18.609860 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:18.611773 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:18.613619 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.613627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.613630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.614042 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.614264 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.614265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.614266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.614437 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.614697 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:18.614703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:18.614936 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 15:25:18.615088 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 15:25:18.615375 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":145,"threads_started":1}
I20260925 15:25:18.616267 4434 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37553
I20260925 15:25:18.616398 4434 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:18.616648 4434 heartbeater.cc:511] Master 127.1.163.126:37553 requested a full tablet report, sending...
I20260925 15:25:18.617125 4109 ts_manager.cc:194] Registered new tserver with Master: 9bfdce6e42d34f93900507086b32ad88 (127.1.163.66:34327)
I20260925 15:25:18.617530 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:59537
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:18.640635 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:18.724848 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:18.725073 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:18.725098 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:18.726356 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:18.726402 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:18.726431 4439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:18.727955 4439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:18.728225 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:18.728474 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:18.733785 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.733779 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:18.733803 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:18.733986 4439 server_base.cc:1034] running on GCE node
I20260925 15:25:18.734218 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:18.734475 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:18.735646 4439 hybrid_clock.cc:648] HybridClock initialized: now 1790349918735622 us; error 44 us; skew 500 ppm
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:18.739112 4439 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:18.739770 4439 webserver.cc:460] Webserver started at http://127.1.163.67:42621/ using document root <none> and password file <none>
I20260925 15:25:18.740083 4439 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:18.740170 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:18.740324 4439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:18.741813 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "aa74c33198c44e1ebdc276326a04b98a"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "bb1ccc9d5e8aab91658ec1e0acf04fd8"
server_key_iv: "e9f98d6ada923f8d9db87b61b1643de4"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.742283 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "aa74c33198c44e1ebdc276326a04b98a"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "bb1ccc9d5e8aab91658ec1e0acf04fd8"
server_key_iv: "e9f98d6ada923f8d9db87b61b1643de4"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.743870 4439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:18.744774 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.744968 4439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:18.745078 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "aa74c33198c44e1ebdc276326a04b98a"
format_stamp: "Formatted at 2026-09-25 15:25:18 on dist-test-slave-hvv5"
server_key: "bb1ccc9d5e8aab91658ec1e0acf04fd8"
server_key_iv: "e9f98d6ada923f8d9db87b61b1643de4"
server_key_version: "encryptionkey@0"
I20260925 15:25:18.745173 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:18.761380 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:18.763450 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:18.763628 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:18.763878 4439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:18.764218 4439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:18.764281 4439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.764334 4439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:18.764365 4439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:18.770435 4439 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:37233
I20260925 15:25:18.770483 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:37233 every 8 connection(s)
I20260925 15:25:18.770798 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:18.776304 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4439
I20260925 15:25:18.776400 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:18.776614 1677 external_mini_cluster.cc:1428] Setting key 9136e6b774a081bb4fa4ebca86da65f2
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349918, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:18.781294 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.771543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:52797 (local address 127.1.163.126:37553)
0925 15:25:18.771616 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:18.771618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:18.771868 (+ 250us) server_negotiation.cc:407] Connection header received
0925 15:25:18.772422 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:18.772424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:18.772434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:18.772504 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:18.773592 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.774115 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.775276 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.775467 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.775528 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.778309 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:18.778311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:18.778311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:18.778317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:18.779822 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.780281 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.780282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.780283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.780346 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:18.780640 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:18.780641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:18.780642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:18.780765 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:18.780825 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:18.781152 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:25:18.781229 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:18.781492 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:18.771542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37553 (local address 127.1.163.67:52797)
0925 15:25:18.771713 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:18.771755 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:18.772261 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:18.772641 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:18.772645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:18.773266 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:18.773502 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.773508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.774228 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.774229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.775184 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:18.775187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:18.775553 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:18.775553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:18.775659 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:18.776208 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:18.776217 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:18.778222 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:18.779921 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.779928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.779930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.780182 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.780432 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:18.780433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:18.780434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:18.780571 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:18.780851 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:18.780857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:18.780991 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:25:18.781080 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:25:18.781212 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260925 15:25:18.782024 4568 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37553
I20260925 15:25:18.782133 4568 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:18.782327 4568 heartbeater.cc:511] Master 127.1.163.126:37553 requested a full tablet report, sending...
I20260925 15:25:18.782852 4109 ts_manager.cc:194] Registered new tserver with Master: aa74c33198c44e1ebdc276326a04b98a (127.1.163.67:37233)
I20260925 15:25:18.783283 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:52797
I20260925 15:25:18.788914 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:25:18.800482 4152 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:18.799365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.1.163.126:37553)
0925 15:25:18.799439 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:18.799441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:18.799569 (+ 128us) server_negotiation.cc:407] Connection header received
0925 15:25:18.799634 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:18.799636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:18.799646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:18.799743 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:18.800359 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41572: BlockingRecv error: recv got EOF from 127.0.0.1:41572 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:25:18.802003 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4171
I20260925 15:25:18.807128 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4305
I20260925 15:25:18.811650 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4439
I20260925 15:25:18.815898 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4077
2026-09-25T15:25:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1917 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4580](info): set up 2 sockets
Sep 25 15:25:18 dist-test-slave-hvv5 krb5kdc[4580](info): commencing operation
krb5kdc: starting...
W20260925 15:25:19.896525 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:19 dist-test-slave-hvv5 krb5kdc[4580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:20.020192 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43641
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:43641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:20.105859 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:20.106146 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:20.106195 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:20.107609 4593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:20.107689 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:20.107718 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:20.107754 4593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:20.107797 4593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:20.109656 4593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:43641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:20.110001 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:20.110276 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:20.115947 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:20.116195 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:20.116233 4593 server_base.cc:1034] running on GCE node
W20260925 15:25:20.116051 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:20.116628 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:20.116951 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:20.118125 4593 hybrid_clock.cc:648] HybridClock initialized: now 1790349920118114 us; error 49 us; skew 500 ppm
Sep 25 15:25:20 dist-test-slave-hvv5 krb5kdc[4580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:20.120796 4593 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:20.121268 4593 webserver.cc:460] Webserver started at http://127.1.163.126:34673/ using document root <none> and password file <none>
I20260925 15:25:20.121511 4593 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:20.121559 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:20.121718 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:20.122840 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "f771f4afc61c47928806d25f7e27504c"
format_stamp: "Formatted at 2026-09-25 15:25:20 on dist-test-slave-hvv5"
server_key: "424a72793cc2a34a1da990e1684705d6"
server_key_iv: "369072d6d4f10ec3a0af5016e58e72f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:20.123209 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "f771f4afc61c47928806d25f7e27504c"
format_stamp: "Formatted at 2026-09-25 15:25:20 on dist-test-slave-hvv5"
server_key: "424a72793cc2a34a1da990e1684705d6"
server_key_iv: "369072d6d4f10ec3a0af5016e58e72f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:20.124727 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 15:25:20.125722 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:20.126011 4593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:20.126132 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "f771f4afc61c47928806d25f7e27504c"
format_stamp: "Formatted at 2026-09-25 15:25:20 on dist-test-slave-hvv5"
server_key: "424a72793cc2a34a1da990e1684705d6"
server_key_iv: "369072d6d4f10ec3a0af5016e58e72f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:20.126224 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:20.136866 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:20.139295 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:20.139467 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:20.143397 4593 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:43641
I20260925 15:25:20.143437 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:43641 every 8 connection(s)
I20260925 15:25:20.143821 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:20.144788 4661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:20.145361 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4593
I20260925 15:25:20.145481 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:20.145658 1677 external_mini_cluster.cc:1428] Setting key 6860585316e889603783bacb426d2ffc
I20260925 15:25:20.148011 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: Bootstrap starting.
I20260925 15:25:20.148787 4661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:20.149091 4661 log.cc:824] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:20.149845 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: No bootstrap required, opened a new log
I20260925 15:25:20.152423 4661 raft_consensus.cc:348] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } }
I20260925 15:25:20.152654 4661 raft_consensus.cc:374] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:20.152710 4661 raft_consensus.cc:729] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f771f4afc61c47928806d25f7e27504c, State: Initialized, Role: FOLLOWER
I20260925 15:25:20.152854 4661 consensus_queue.cc:260] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } }
I20260925 15:25:20.152947 4661 raft_consensus.cc:388] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:20.152987 4661 raft_consensus.cc:482] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:20.153044 4661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:25:20 dist-test-slave-hvv5 krb5kdc[4580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:20.153893 4661 raft_consensus.cc:504] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } }
I20260925 15:25:20.154026 4661 leader_election.cc:302] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f771f4afc61c47928806d25f7e27504c; no voters:
I20260925 15:25:20.154235 4661 leader_election.cc:288] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:20.154310 4666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:20.154528 4661 sys_catalog.cc:564] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:20.154536 4666 raft_consensus.cc:686] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [term 1 LEADER]: Becoming Leader. State: Replica: f771f4afc61c47928806d25f7e27504c, State: Running, Role: LEADER
I20260925 15:25:20.154724 4666 consensus_queue.cc:237] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } }
I20260925 15:25:20.155208 4666 sys_catalog.cc:455] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f771f4afc61c47928806d25f7e27504c. Latest consensus state: current_term: 1 leader_uuid: "f771f4afc61c47928806d25f7e27504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } } }
I20260925 15:25:20.155326 4666 sys_catalog.cc:458] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:20.157232 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f771f4afc61c47928806d25f7e27504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f771f4afc61c47928806d25f7e27504c" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43641 } } }
I20260925 15:25:20.157366 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c [sys.catalog]: This master's current role is: LEADER
W20260925 15:25:20.157025 4681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:25:20.157806 4681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:25:20.157863 4675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:20.158126 4675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:20.159871 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:20.146224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.1.163.126:43641)
0925 15:25:20.146417 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:20.146427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:20.146473 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:25:20.147039 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:20.147048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:20.147113 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:20.147291 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:20.147874 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:20.149316 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:20.150201 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:20.150832 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:20.150955 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:20.154199 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:20.154203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:20.154203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:20.154235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:20.157026 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:20.157561 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:20.157565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:20.157565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:20.157683 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:20.158428 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:20.158430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:20.158430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:20.158763 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:20.158926 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:20.159498 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 15:25:20.159657 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260925 15:25:20.160790 4675 catalog_manager.cc:1347] Generated new cluster ID: c373a9a9574d409c96b33811bbb7ec1f
I20260925 15:25:20.160840 4675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:20.164775 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:25:20.167645 4675 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:20.168272 4675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:20.182387 4675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f771f4afc61c47928806d25f7e27504c: Generated new TSK 0
I20260925 15:25:20.182612 4675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:20.190918 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:20.186912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.1.163.126:43641)
0925 15:25:20.186979 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:20.186981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:20.187087 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:25:20.187158 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:20.187159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:20.187173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:20.187248 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:20.187620 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:20.188396 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:20.189230 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:20.189570 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:20.189652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:20.190356 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:20.190358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:20.190358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:20.190629 (+ 271us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:20.190649 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:20.190657 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:20.190727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:20.190794 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:25:20.190826 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:25:20.193609 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4593
2026-09-25T15:25:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1373 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:20 dist-test-slave-hvv5 krb5kdc[4694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:20 dist-test-slave-hvv5 krb5kdc[4694](info): set up 2 sockets
Sep 25 15:25:20 dist-test-slave-hvv5 krb5kdc[4694](info): commencing operation
krb5kdc: starting...
W20260925 15:25:21.224860 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:21.352026 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36353
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:21.434602 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:21.434837 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:21.434862 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:21.436038 4707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:21.436086 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:21.436102 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:21.436115 4707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:21.436127 4707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:21.437835 4707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:21.438087 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:21.438308 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:21.444036 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.444039 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.444039 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:21.444072 4707 server_base.cc:1034] running on GCE node
I20260925 15:25:21.444748 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:21.445079 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:21.446259 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790349921446231 us; error 36 us; skew 500 ppm
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:21.449194 4707 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:21.449721 4707 webserver.cc:460] Webserver started at http://127.1.163.126:34471/ using document root <none> and password file <none>
I20260925 15:25:21.449996 4707 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:21.450073 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:21.450222 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:21.451328 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "cb777bd41a4745929f95de5806c647de"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "6a565c59e05a513798c5335f01476f05"
server_key_iv: "9d9d5d3e0cd9a9878a00238dc99ae27d"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.451702 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "cb777bd41a4745929f95de5806c647de"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "6a565c59e05a513798c5335f01476f05"
server_key_iv: "9d9d5d3e0cd9a9878a00238dc99ae27d"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.453401 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:21.454347 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.454551 4707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:21.454676 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "cb777bd41a4745929f95de5806c647de"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "6a565c59e05a513798c5335f01476f05"
server_key_iv: "9d9d5d3e0cd9a9878a00238dc99ae27d"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.454790 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:21.472855 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:21.475312 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:21.475508 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:21.479728 4707 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36353
I20260925 15:25:21.479759 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36353 every 8 connection(s)
I20260925 15:25:21.480145 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:21.481359 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:21.483700 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: Bootstrap starting.
I20260925 15:25:21.484333 4775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:21.484692 4775 log.cc:824] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:21.485437 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: No bootstrap required, opened a new log
I20260925 15:25:21.487360 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4707
I20260925 15:25:21.487478 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:21.487315 4775 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } }
I20260925 15:25:21.487514 4775 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:21.487547 4775 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb777bd41a4745929f95de5806c647de, State: Initialized, Role: FOLLOWER
I20260925 15:25:21.487669 1677 external_mini_cluster.cc:1428] Setting key 407c7673ca707b1db2ef19752b6d452f
I20260925 15:25:21.487705 4775 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } }
I20260925 15:25:21.487787 4775 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:21.487815 4775 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:21.487861 4775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:21.489158 4775 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } }
I20260925 15:25:21.489290 4775 leader_election.cc:302] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb777bd41a4745929f95de5806c647de; no voters:
I20260925 15:25:21.489472 4775 leader_election.cc:288] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:21.489539 4780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:21.489745 4775 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:21.489770 4780 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [term 1 LEADER]: Becoming Leader. State: Replica: cb777bd41a4745929f95de5806c647de, State: Running, Role: LEADER
I20260925 15:25:21.489965 4780 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } }
I20260925 15:25:21.490381 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb777bd41a4745929f95de5806c647de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } } }
I20260925 15:25:21.490475 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:21.490617 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb777bd41a4745929f95de5806c647de. Latest consensus state: current_term: 1 leader_uuid: "cb777bd41a4745929f95de5806c647de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb777bd41a4745929f95de5806c647de" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36353 } } }
I20260925 15:25:21.490660 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de [sys.catalog]: This master's current role is: LEADER
W20260925 15:25:21.491747 4796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:25:21.491809 4796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:25:21.491858 4789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:21.492012 4789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:21.493731 4789 catalog_manager.cc:1347] Generated new cluster ID: 639fa648c64a4af6b8a703c25a999e55
I20260925 15:25:21.493816 4789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:21.499974 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.488222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.1.163.126:36353)
0925 15:25:21.488466 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:21.488479 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:21.488524 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:25:21.488727 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:21.488734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:21.488796 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:21.488956 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:21.489690 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.491175 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.492206 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.492859 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.493100 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:21.495982 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:21.495987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:21.495987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:21.496001 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:21.498172 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.498600 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.498603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.498604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.498762 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.498992 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.498993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.498993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.499342 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:21.499523 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:21.499600 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:25:21.499752 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
I20260925 15:25:21.513115 4789 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:21.513870 4789 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:21.518832 4789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb777bd41a4745929f95de5806c647de: Generated new TSK 0
I20260925 15:25:21.519063 4789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:21.548244 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:21.633211 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:21.633432 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:21.633458 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:21.634716 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:21.634766 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:21.634796 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:21.636360 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:21.636673 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:21.636931 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:21.642162 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.642235 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.642184 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:21.642554 4801 server_base.cc:1034] running on GCE node
I20260925 15:25:21.642736 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:21.642993 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:21.644193 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790349921644180 us; error 38 us; skew 500 ppm
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:21.647037 4801 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:21.647523 4801 webserver.cc:460] Webserver started at http://127.1.163.65:42163/ using document root <none> and password file <none>
I20260925 15:25:21.647758 4801 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:21.647828 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:21.647985 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:21.649163 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "1a13ebe771424d95ab02bbe969adf7b4"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "66d9aceecf88159a658cc203363f59c3"
server_key_iv: "1c03eafa8a2b5163adc040af68d489bf"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.649546 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "1a13ebe771424d95ab02bbe969adf7b4"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "66d9aceecf88159a658cc203363f59c3"
server_key_iv: "1c03eafa8a2b5163adc040af68d489bf"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.651023 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:21.651893 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.652102 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:21.652209 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "1a13ebe771424d95ab02bbe969adf7b4"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "66d9aceecf88159a658cc203363f59c3"
server_key_iv: "1c03eafa8a2b5163adc040af68d489bf"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.652287 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:21.661701 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:21.664088 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:21.664261 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:21.664525 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:21.665014 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:21.665071 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.665114 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:21.665143 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.671175 4801 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:40151
I20260925 15:25:21.671221 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:40151 every 8 connection(s)
I20260925 15:25:21.671550 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:21.673563 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4801
I20260925 15:25:21.673687 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:21.673930 1677 external_mini_cluster.cc:1428] Setting key 4cf386c4e5a23fb04fa6e8291c1573e9
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:21.683069 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.672317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:43079 (local address 127.1.163.126:36353)
0925 15:25:21.672419 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:21.672420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:21.672614 (+ 194us) server_negotiation.cc:407] Connection header received
0925 15:25:21.673152 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:21.673153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:21.673168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:21.673259 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:21.674817 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.675333 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.676765 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.677013 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.677071 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:21.679642 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:21.679645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:21.679645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:21.679655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:21.681505 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.681976 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.681978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.681978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.682044 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.682335 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.682336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.682336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.682496 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:21.682574 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:21.682861 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:25:21.682959 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:25:21.683100 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.672285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36353 (local address 127.1.163.65:43079)
0925 15:25:21.672443 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:21.672488 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:21.673008 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:21.673464 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:21.673469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:21.674256 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:21.674562 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.674568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.675451 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.675452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.676660 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.676663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.677105 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.677106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.677258 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:21.677822 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:21.677832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:21.679539 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:21.681616 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:21.681622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:21.681624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:21.681892 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:21.682143 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:21.682144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:21.682144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:21.682255 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:21.682586 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:21.682591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:21.682712 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 15:25:21.682793 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:25:21.682927 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:21.683676 4930 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36353
I20260925 15:25:21.683791 4930 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:21.684011 4930 heartbeater.cc:511] Master 127.1.163.126:36353 requested a full tablet report, sending...
I20260925 15:25:21.684685 4739 ts_manager.cc:194] Registered new tserver with Master: 1a13ebe771424d95ab02bbe969adf7b4 (127.1.163.65:40151)
I20260925 15:25:21.685360 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:43079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:21.704042 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:21.791853 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:21.792095 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:21.792136 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:21.793467 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:21.793550 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:21.793601 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:21.795305 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:21.795589 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:21.795830 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:21.801510 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.801502 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.801502 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:21.801582 4935 server_base.cc:1034] running on GCE node
I20260925 15:25:21.802186 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:21.802452 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:21.803624 4935 hybrid_clock.cc:648] HybridClock initialized: now 1790349921803612 us; error 31 us; skew 500 ppm
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:21.806648 4935 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:21.807250 4935 webserver.cc:460] Webserver started at http://127.1.163.66:41675/ using document root <none> and password file <none>
I20260925 15:25:21.807464 4935 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:21.807508 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:21.807601 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:21.808735 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "d5aa95bfb6734367909d1ea39f5e1dfd"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "5d531d3756f53f07e13a3765ade0888c"
server_key_iv: "d5ec0e18773ea56008b52be6d6550b69"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.809054 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "d5aa95bfb6734367909d1ea39f5e1dfd"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "5d531d3756f53f07e13a3765ade0888c"
server_key_iv: "d5ec0e18773ea56008b52be6d6550b69"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.810541 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:25:21.811327 4950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.811551 4935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:21.811632 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "d5aa95bfb6734367909d1ea39f5e1dfd"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "5d531d3756f53f07e13a3765ade0888c"
server_key_iv: "d5ec0e18773ea56008b52be6d6550b69"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.811728 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:21.821593 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:21.823951 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:21.824126 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:21.824391 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:21.824800 4935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:21.824906 4935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.824954 4935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:21.824980 4935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.831336 4935 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:33637
I20260925 15:25:21.831393 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:33637 every 8 connection(s)
I20260925 15:25:21.831810 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:21.840010 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 4935
I20260925 15:25:21.840131 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:21.840353 1677 external_mini_cluster.cc:1428] Setting key 7779371d7cdf152dcb101d4f87caa2a6
I20260925 15:25:21.842047 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.832766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:58959 (local address 127.1.163.126:36353)
0925 15:25:21.832874 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:21.832876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:21.833077 (+ 201us) server_negotiation.cc:407] Connection header received
0925 15:25:21.833624 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:21.833626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:21.833638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:21.833692 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:21.834736 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.835253 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.836305 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.836515 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.836672 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:21.838959 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:21.838961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:21.838961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:21.838970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:21.840616 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.841042 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.841044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.841044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.841108 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:21.841364 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:21.841365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:21.841365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:21.841494 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:21.841574 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:21.841854 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:25:21.841944 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:25:21.842427 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.832680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36353 (local address 127.1.163.66:58959)
0925 15:25:21.832938 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:21.832983 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:21.833513 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:21.833782 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:21.833785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:21.834419 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:21.834630 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.834635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.835355 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.835356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.836191 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.836193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.836680 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.836682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.836808 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:21.837385 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:21.837394 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:21.838847 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:21.840716 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:21.840723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:21.840725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:21.840947 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:21.841182 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:21.841183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:21.841183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:21.841293 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:21.841569 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:21.841576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:21.841713 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:25:21.841821 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:25:21.841989 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260925 15:25:21.843206 5064 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36353
I20260925 15:25:21.843323 5064 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:21.843554 5064 heartbeater.cc:511] Master 127.1.163.126:36353 requested a full tablet report, sending...
I20260925 15:25:21.843961 4739 ts_manager.cc:194] Registered new tserver with Master: d5aa95bfb6734367909d1ea39f5e1dfd (127.1.163.66:33637)
I20260925 15:25:21.844362 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:58959
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:21.867580 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:21.951731 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:21.951984 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:21.952009 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:21.953306 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:21.953353 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:21.953380 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:21.955019 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:21.955277 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:21.955549 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:21.961360 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:21.961519 5069 server_base.cc:1034] running on GCE node
W20260925 15:25:21.961371 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:21.961400 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:21.961796 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:21.962065 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:21.963292 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790349921963289 us; error 44 us; skew 500 ppm
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:21.966111 5069 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:21.966594 5069 webserver.cc:460] Webserver started at http://127.1.163.67:34735/ using document root <none> and password file <none>
I20260925 15:25:21.966845 5069 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:21.966929 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:21.967079 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:21.968155 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "5cf62ea6558d4b018108aebaa4cb382f"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "8403914c566db0678a38bf29e2c7c5fb"
server_key_iv: "ab3cd19ae01c2938053e215cdf9eb574"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.968530 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "5cf62ea6558d4b018108aebaa4cb382f"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "8403914c566db0678a38bf29e2c7c5fb"
server_key_iv: "ab3cd19ae01c2938053e215cdf9eb574"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.970079 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:21.970919 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.971112 5069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:21.971217 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "5cf62ea6558d4b018108aebaa4cb382f"
format_stamp: "Formatted at 2026-09-25 15:25:21 on dist-test-slave-hvv5"
server_key: "8403914c566db0678a38bf29e2c7c5fb"
server_key_iv: "ab3cd19ae01c2938053e215cdf9eb574"
server_key_version: "encryptionkey@0"
I20260925 15:25:21.971316 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:21.984197 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:21.986433 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:21.986639 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:21.986928 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:21.987293 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:21.987345 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.987381 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:21.987413 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:21.993871 5069 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:37063
I20260925 15:25:21.993911 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:37063 every 8 connection(s)
I20260925 15:25:21.994287 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:21 dist-test-slave-hvv5 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:22.003170 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5069
I20260925 15:25:22.003278 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:22.003500 1677 external_mini_cluster.cc:1428] Setting key ae29bb667c479a4da0129503c8edefd1
I20260925 15:25:22.005261 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.995094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34509 (local address 127.1.163.126:36353)
0925 15:25:21.995157 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:21.995159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:21.995556 (+ 397us) server_negotiation.cc:407] Connection header received
0925 15:25:21.996074 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:21.996075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:21.996087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:21.996148 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:21.997240 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.997849 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.998939 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.999148 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.999211 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:22.001650 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:22.001651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:22.001651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:22.001659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:22.003426 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:22.003925 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:22.003927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:22.003927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:22.004009 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:22.004501 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:22.004501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:22.004502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:22.004707 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:22.004795 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:22.005120 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:25:22.005192 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:22.005482 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:21.995092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36353 (local address 127.1.163.67:34509)
0925 15:25:21.995369 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:21.995420 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:21.995959 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:21.996205 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:21.996209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:21.996926 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:21.997132 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.997137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.997953 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.997954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.998846 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:21.998850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:21.999255 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:21.999256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:21.999411 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:22.000057 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:22.000066 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:22.001537 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:22.003513 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:22.003522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:22.003525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:22.003843 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:22.004109 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:22.004111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:22.004111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:22.004411 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:22.004782 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:22.004789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:22.004982 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 15:25:22.005077 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:25:22.005224 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260925 15:25:22.006106 5198 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36353
I20260925 15:25:22.006263 5198 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:22.006472 5198 heartbeater.cc:511] Master 127.1.163.126:36353 requested a full tablet report, sending...
I20260925 15:25:22.006908 4739 ts_manager.cc:194] Registered new tserver with Master: 5cf62ea6558d4b018108aebaa4cb382f (127.1.163.67:37063)
I20260925 15:25:22.007339 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:34509
I20260925 15:25:22.016034 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:25:22 dist-test-slave-hvv5 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349922, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:25:22 dist-test-slave-hvv5 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349922, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:22.040313 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:22.032238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.1.163.126:36353)
0925 15:25:22.032328 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:22.032331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:22.032427 (+ 96us) server_negotiation.cc:407] Connection header received
0925 15:25:22.032528 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:22.032532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:22.032573 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:22.032624 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:22.033259 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:22.033763 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:22.034544 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:22.034739 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:22.034831 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:22.037322 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:22.037324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:22.037324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:22.037331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:22.038973 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:22.039314 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:22.039315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:22.039315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:22.039376 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:22.039594 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:22.039595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:22.039595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:22.039750 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:22.039840 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:22.040099 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:25:22.040189 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 15:25:22.040627 4739 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55006
I20260925 15:25:22.041185 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4801
I20260925 15:25:22.046150 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4935
I20260925 15:25:22.050621 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5069
I20260925 15:25:22.054797 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 4707
2026-09-25T15:25:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1863 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:22 dist-test-slave-hvv5 krb5kdc[5210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:22 dist-test-slave-hvv5 krb5kdc[5210](info): set up 2 sockets
Sep 25 15:25:22 dist-test-slave-hvv5 krb5kdc[5210](info): commencing operation
krb5kdc: starting...
W20260925 15:25:23.112915 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:23.243757 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43353
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:43353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:23.328184 5223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:23.328464 5223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:23.328505 5223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:23.329777 5223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:23.329865 5223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:23.329896 5223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:23.329933 5223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:23.329970 5223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:23.331589 5223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:43353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:23.331900 5223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:23.332150 5223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:23.337477 5228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.337468 5231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.337491 5229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:23.337882 5223 server_base.cc:1034] running on GCE node
I20260925 15:25:23.338145 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:23.338471 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:23.339659 5223 hybrid_clock.cc:648] HybridClock initialized: now 1790349923339644 us; error 49 us; skew 500 ppm
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:23.347103 5223 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:23.347676 5223 webserver.cc:460] Webserver started at http://127.1.163.126:34007/ using document root <none> and password file <none>
I20260925 15:25:23.347929 5223 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:23.348001 5223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:23.348163 5223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:23.349371 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "8d5dbb761ed14de4baece6f9e7c77666"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "62fe7902ef728a997d5fa79d7c293c98"
server_key_iv: "85987a255f8d92ddcef19520aaca4acc"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.349753 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "8d5dbb761ed14de4baece6f9e7c77666"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "62fe7902ef728a997d5fa79d7c293c98"
server_key_iv: "85987a255f8d92ddcef19520aaca4acc"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.351190 5223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:25:23.352021 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.352238 5223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:23.352412 5223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "8d5dbb761ed14de4baece6f9e7c77666"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "62fe7902ef728a997d5fa79d7c293c98"
server_key_iv: "85987a255f8d92ddcef19520aaca4acc"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.352501 5223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:23.402786 5223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:23.405272 5223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:23.405474 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:23.409605 5223 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:43353
I20260925 15:25:23.409663 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:43353 every 8 connection(s)
I20260925 15:25:23.410233 5223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:23.411007 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:23.413332 5291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666: Bootstrap starting.
I20260925 15:25:23.414081 5291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:23.414431 5291 log.cc:824] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:23.415205 5291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666: No bootstrap required, opened a new log
I20260925 15:25:23.416986 5291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } }
I20260925 15:25:23.417132 5291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:23.417155 5291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5dbb761ed14de4baece6f9e7c77666, State: Initialized, Role: FOLLOWER
I20260925 15:25:23.417315 5291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } }
I20260925 15:25:23.417397 5291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:23.417420 5291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:23.417490 5291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:23.418414 5291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } }
I20260925 15:25:23.418502 5291 leader_election.cc:302] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d5dbb761ed14de4baece6f9e7c77666; no voters:
I20260925 15:25:23.418725 5291 leader_election.cc:288] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:23.418771 5294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:23.418988 5294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [term 1 LEADER]: Becoming Leader. State: Replica: 8d5dbb761ed14de4baece6f9e7c77666, State: Running, Role: LEADER
I20260925 15:25:23.419015 5291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:23.419091 5294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } }
I20260925 15:25:23.419543 5296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d5dbb761ed14de4baece6f9e7c77666. Latest consensus state: current_term: 1 leader_uuid: "8d5dbb761ed14de4baece6f9e7c77666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } } }
I20260925 15:25:23.419631 5296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:23.419704 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5223
I20260925 15:25:23.419812 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:23.420024 1677 external_mini_cluster.cc:1428] Setting key 48d45328c558a0b357758db7560316b2
I20260925 15:25:23.420360 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d5dbb761ed14de4baece6f9e7c77666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5dbb761ed14de4baece6f9e7c77666" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43353 } } }
I20260925 15:25:23.420418 5303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:23.420439 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:23.421036 5303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:23.422866 5303 catalog_manager.cc:1347] Generated new cluster ID: 62ad8ade6a1c4d1994f191c51e2ea7b9
I20260925 15:25:23.422921 5303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:23.430804 5303 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:23.431530 5303 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:23.432644 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.420640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.1.163.126:43353)
0925 15:25:23.421164 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:23.421174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:23.421204 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:25:23.421326 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:23.421333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:23.421391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:23.421534 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:23.422339 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.423747 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.424856 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.425504 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.425626 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.428510 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:23.428514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:23.428515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:23.428528 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:23.430608 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.431051 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.431055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.431055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.431168 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.431375 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.431376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.431376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.431630 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:23.432168 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:23.432235 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:25:23.432381 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":84,"threads_started":1}
I20260925 15:25:23.445693 5303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d5dbb761ed14de4baece6f9e7c77666: Generated new TSK 0
I20260925 15:25:23.445942 5303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:23.473987 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:23.559769 5317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:23.560027 5317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:23.560071 5317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:23.561348 5317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:23.561427 5317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:23.561476 5317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:23.563156 5317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:23.563457 5317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:23.563717 5317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:23.569461 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.569461 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.569520 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:23.569792 5317 server_base.cc:1034] running on GCE node
I20260925 15:25:23.570093 5317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:23.570524 5317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:23.571720 5317 hybrid_clock.cc:648] HybridClock initialized: now 1790349923571707 us; error 61 us; skew 500 ppm
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:23.574738 5317 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:23.575340 5317 webserver.cc:460] Webserver started at http://127.1.163.65:44217/ using document root <none> and password file <none>
I20260925 15:25:23.575603 5317 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:23.575675 5317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:23.575845 5317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:23.577049 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "eee0a945fc2e4dbbb2dc9591c8e160d4"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "249ab07505abb413873de11403670c5d"
server_key_iv: "9d5aa1e39fddcf99f23ec40485b288c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.577433 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "eee0a945fc2e4dbbb2dc9591c8e160d4"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "249ab07505abb413873de11403670c5d"
server_key_iv: "9d5aa1e39fddcf99f23ec40485b288c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.578995 5317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:23.579897 5332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.580137 5317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:23.580269 5317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "eee0a945fc2e4dbbb2dc9591c8e160d4"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "249ab07505abb413873de11403670c5d"
server_key_iv: "9d5aa1e39fddcf99f23ec40485b288c7"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.580359 5317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:23.615139 5317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:23.617878 5317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:23.618288 5317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:23.618647 5317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:23.619135 5317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:23.619216 5317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.619269 5317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:23.619300 5317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.626009 5317 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:42923
I20260925 15:25:23.626050 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:42923 every 8 connection(s)
I20260925 15:25:23.626413 5317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:23.630488 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5317
I20260925 15:25:23.630577 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:23.630796 1677 external_mini_cluster.cc:1428] Setting key 0eb09a5f2f819e39ad17cb3e294d2677
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:23.638252 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.627334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:51923 (local address 127.1.163.126:43353)
0925 15:25:23.627404 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:23.627406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:23.627618 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:25:23.628638 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:23.628639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:23.628653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:23.628741 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:23.629449 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.629989 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.631335 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.631616 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.631824 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.634534 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:23.634536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:23.634536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:23.634545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:23.636463 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.637013 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.637015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.637015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.637093 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.637450 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.637451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.637452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.637629 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:23.637712 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:23.638045 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 15:25:23.638144 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:25:23.638334 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.627243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43353 (local address 127.1.163.65:51923)
0925 15:25:23.627474 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:23.627525 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:23.628470 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:23.628854 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:23.628857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:23.629125 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:23.629335 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.629340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.630111 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.630116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.631202 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.631205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.631864 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.631866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.632025 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.632720 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:23.632732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:23.634416 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:23.636637 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.636644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.636647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.636914 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.637231 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.637232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.637232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.637361 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.637721 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:23.637728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:23.637873 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:25:23.637968 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:25:23.638110 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260925 15:25:23.638926 5446 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43353
I20260925 15:25:23.639046 5446 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:23.639262 5446 heartbeater.cc:511] Master 127.1.163.126:43353 requested a full tablet report, sending...
I20260925 15:25:23.639918 5255 ts_manager.cc:194] Registered new tserver with Master: eee0a945fc2e4dbbb2dc9591c8e160d4 (127.1.163.65:42923)
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:23.640928 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:51923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:23.660923 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:23.745994 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:23.746258 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:23.746300 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:23.747680 5451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:23.747762 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:23.747815 5451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:23.749629 5451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:23.749948 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:23.750217 5451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:23.755880 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.755981 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.755883 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:23.756090 5451 server_base.cc:1034] running on GCE node
I20260925 15:25:23.756446 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:23.756726 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:23.757954 5451 hybrid_clock.cc:648] HybridClock initialized: now 1790349923757900 us; error 36 us; skew 500 ppm
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:23.761627 5451 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:23.762115 5451 webserver.cc:460] Webserver started at http://127.1.163.66:44677/ using document root <none> and password file <none>
I20260925 15:25:23.762362 5451 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:23.762439 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:23.762606 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:23.763736 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "800b64a5e12b49438645ade50a211bee"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "3adb2c69146994f77562cf72ae8ff409"
server_key_iv: "d5781b067076c8c189e8513a2ec60c4d"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.764127 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "800b64a5e12b49438645ade50a211bee"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "3adb2c69146994f77562cf72ae8ff409"
server_key_iv: "d5781b067076c8c189e8513a2ec60c4d"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.765738 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:23.766673 5466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.766876 5451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:23.766988 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "800b64a5e12b49438645ade50a211bee"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "3adb2c69146994f77562cf72ae8ff409"
server_key_iv: "d5781b067076c8c189e8513a2ec60c4d"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.767127 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:23.791690 5451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:23.794013 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:23.794204 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:23.794476 5451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:23.794842 5451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:23.794955 5451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.794998 5451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:23.795028 5451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.800838 5451 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:40435
I20260925 15:25:23.800948 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:40435 every 8 connection(s)
I20260925 15:25:23.801244 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:23.806667 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5451
I20260925 15:25:23.806789 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:23.806981 1677 external_mini_cluster.cc:1428] Setting key 10f106433e43bedd5f48e55884a5de23
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:23.812603 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.802022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:36269 (local address 127.1.163.126:43353)
0925 15:25:23.802111 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:23.802113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:23.802300 (+ 187us) server_negotiation.cc:407] Connection header received
0925 15:25:23.803332 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:23.803334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:23.803349 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:23.803397 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:23.804063 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.804684 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.805722 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.805925 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.806017 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.809008 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:23.809010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:23.809010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:23.809025 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:23.810833 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.811347 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.811348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.811349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.811419 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.811761 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.811762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.811762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.811941 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:23.812021 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:23.812343 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:25:23.812436 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:23.812708 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.801997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43353 (local address 127.1.163.66:36269)
0925 15:25:23.802171 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:23.802218 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:23.803197 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:23.803472 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:23.803476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:23.803749 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:23.803951 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.803955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.804778 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.804779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.805612 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.805614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.806040 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.806041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.806167 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.806945 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:23.806954 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:23.808897 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:23.810979 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.810986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.810999 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.811259 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.811517 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.811518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.811519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.811635 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.812051 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:23.812057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:23.812192 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:25:23.812295 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:25:23.812442 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260925 15:25:23.813267 5580 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43353
I20260925 15:25:23.813400 5580 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:23.813629 5580 heartbeater.cc:511] Master 127.1.163.126:43353 requested a full tablet report, sending...
I20260925 15:25:23.814123 5255 ts_manager.cc:194] Registered new tserver with Master: 800b64a5e12b49438645ade50a211bee (127.1.163.66:40435)
I20260925 15:25:23.814692 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:36269
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:23.836648 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:23.922927 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:23.923175 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:23.923203 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:23.924520 5585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:23.924634 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:23.924666 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:23.926333 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:23.926646 5585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:23.926903 5585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:23.932221 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.932227 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:23.932216 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:23.932631 5585 server_base.cc:1034] running on GCE node
I20260925 15:25:23.932873 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:23.933094 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:23.934258 5585 hybrid_clock.cc:648] HybridClock initialized: now 1790349923934236 us; error 32 us; skew 500 ppm
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:23.937002 5585 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:23.937573 5585 webserver.cc:460] Webserver started at http://127.1.163.67:33907/ using document root <none> and password file <none>
I20260925 15:25:23.937868 5585 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:23.937925 5585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:23.938112 5585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:23.939275 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "a2a0716a2135415096af8bacfd0a8d91"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "cdc2e131c670d742cb0a91171006ea0b"
server_key_iv: "dffc3be09ac5f95abc22ac63b32c02f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.939697 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "a2a0716a2135415096af8bacfd0a8d91"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "cdc2e131c670d742cb0a91171006ea0b"
server_key_iv: "dffc3be09ac5f95abc22ac63b32c02f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.941671 5585 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 15:25:23.942577 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.942833 5585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:23.942912 5585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "a2a0716a2135415096af8bacfd0a8d91"
format_stamp: "Formatted at 2026-09-25 15:25:23 on dist-test-slave-hvv5"
server_key: "cdc2e131c670d742cb0a91171006ea0b"
server_key_iv: "dffc3be09ac5f95abc22ac63b32c02f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:23.943044 5585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:23.952824 5585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:23.955307 5585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:23.955482 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:23.955804 5585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:23.956163 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:23.956199 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.956223 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:23.956235 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:23.962569 5585 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:33643
I20260925 15:25:23.962608 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:33643 every 8 connection(s)
I20260925 15:25:23.962955 5585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:23.972047 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5585
I20260925 15:25:23.972159 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:23.972376 1677 external_mini_cluster.cc:1428] Setting key e7e8cb1bec5afd68e120bb3d3a2cc021
I20260925 15:25:23.973609 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.963782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:45847 (local address 127.1.163.126:43353)
0925 15:25:23.963840 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:23.963842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:23.964000 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:25:23.965022 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:23.965023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:23.965035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:23.965113 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:23.965892 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.966643 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.967730 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.967926 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.967993 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.970371 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:23.970372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:23.970373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:23.970381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:23.971897 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.972398 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.972400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.972400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.972470 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:23.972797 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:23.972798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:23.972799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:23.972931 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:23.973021 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:23.973409 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:25:23.973516 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:23.973711 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:23.963689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43353 (local address 127.1.163.67:45847)
0925 15:25:23.963867 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:23.963911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:23.964890 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:23.965195 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:23.965198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:23.965467 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:23.965726 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.965731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.966727 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.966728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.967643 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:23.967646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:23.968013 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:23.968014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:23.968122 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:23.968752 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:23.968762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:23.970258 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:23.971993 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.972003 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.972005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.972304 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.972584 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:23.972585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:23.972586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:23.972700 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:23.973009 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:23.973015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:23.973210 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 15:25:23.973296 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:25:23.973486 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260925 15:25:23.974267 5714 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43353
I20260925 15:25:23.974359 5714 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:23.974514 5714 heartbeater.cc:511] Master 127.1.163.126:43353 requested a full tablet report, sending...
I20260925 15:25:23.974850 5255 ts_manager.cc:194] Registered new tserver with Master: a2a0716a2135415096af8bacfd0a8d91 (127.1.163.67:33643)
I20260925 15:25:23.975238 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:45847
I20260925 15:25:23.984975 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:25:23 dist-test-slave-hvv5 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:25:24 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:24.009997 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:24.000868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.1.163.65:42923)
0925 15:25:24.001042 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:24.001047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:24.001072 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:25:24.001099 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:24.001101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:24.001116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:24.001176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:24.001778 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:24.002728 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:24.003555 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:24.003877 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:24.003961 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:24.006608 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:24.006620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:24.006620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:24.006627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:24.008428 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:24.008980 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:24.008984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:24.008984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:24.009060 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:24.009327 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:24.009329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:24.009329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:24.009496 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:24.009626 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:24.009702 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:25:24.009771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
Sep 25 15:25:24 dist-test-slave-hvv5 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:24.019230 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:24.011438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.1.163.126:43353)
0925 15:25:24.011497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:24.011499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:24.011508 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:24.011570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:24.011571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:24.011581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:24.011656 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:24.012172 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:24.012744 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:24.013527 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:24.013721 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:24.013809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:24.016258 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:24.016260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:24.016260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:24.016275 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:24.017887 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:24.018333 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:24.018335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:24.018335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:24.018398 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:24.018606 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:24.018607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:24.018607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:24.018776 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:24.018889 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:24.019006 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:25:24.019105 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 15:25:24.019593 5255 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44534
I20260925 15:25:24.020054 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5317
I20260925 15:25:24.024967 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5451
I20260925 15:25:24.029731 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5585
I20260925 15:25:24.034044 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5223
2026-09-25T15:25:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1979 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:24 dist-test-slave-hvv5 krb5kdc[5726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:24 dist-test-slave-hvv5 krb5kdc[5726](info): set up 2 sockets
Sep 25 15:25:24 dist-test-slave-hvv5 krb5kdc[5726](info): commencing operation
krb5kdc: starting...
W20260925 15:25:25.096756 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:25.229547 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:35423
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:35423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:25.315272 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:25.315567 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:25.315614 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:25.316936 5739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:25.317016 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:25.317040 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:25.317071 5739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:25.317102 5739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:25.318778 5739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:35423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:25.319099 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:25.319350 5739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:25.324709 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.324709 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.324718 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:25.325016 5739 server_base.cc:1034] running on GCE node
I20260925 15:25:25.325263 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:25.325584 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:25.326810 5739 hybrid_clock.cc:648] HybridClock initialized: now 1790349925326794 us; error 34 us; skew 500 ppm
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:25.329766 5739 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:25.330130 5739 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:25.330183 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:25.330348 5739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:25.331506 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "d3b7224edb394d2f988e69f5794ca85a"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "51f0087adeada1eaec0f0c05732bdee3"
server_key_iv: "5773b3e061bec6e391fb1c1be4d76a3f"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.331892 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "d3b7224edb394d2f988e69f5794ca85a"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "51f0087adeada1eaec0f0c05732bdee3"
server_key_iv: "5773b3e061bec6e391fb1c1be4d76a3f"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.333370 5739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:25.334167 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.334390 5739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:25.334488 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "d3b7224edb394d2f988e69f5794ca85a"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "51f0087adeada1eaec0f0c05732bdee3"
server_key_iv: "5773b3e061bec6e391fb1c1be4d76a3f"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.334573 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:25.364496 5739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:25.366883 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:25.367064 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:25.371275 5739 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:35423
I20260925 15:25:25.371285 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:35423 every 8 connection(s)
I20260925 15:25:25.371641 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:25.372879 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:25.375385 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: Bootstrap starting.
I20260925 15:25:25.375715 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5739
I20260925 15:25:25.375857 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:25.376091 1677 external_mini_cluster.cc:1428] Setting key 7bda2250f4878bc0c625262f5901f4c9
I20260925 15:25:25.376127 5806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:25.376520 5806 log.cc:824] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:25.377357 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: No bootstrap required, opened a new log
I20260925 15:25:25.379231 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } }
I20260925 15:25:25.379410 5806 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:25.379433 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b7224edb394d2f988e69f5794ca85a, State: Initialized, Role: FOLLOWER
I20260925 15:25:25.379614 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } }
I20260925 15:25:25.379691 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:25.379714 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:25.379779 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:25.380364 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } }
I20260925 15:25:25.380453 5806 leader_election.cc:302] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3b7224edb394d2f988e69f5794ca85a; no voters:
I20260925 15:25:25.380729 5806 leader_election.cc:288] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:25.380756 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:25.380880 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [term 1 LEADER]: Becoming Leader. State: Replica: d3b7224edb394d2f988e69f5794ca85a, State: Running, Role: LEADER
I20260925 15:25:25.380971 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } }
I20260925 15:25:25.381021 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:25.381312 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3b7224edb394d2f988e69f5794ca85a. Latest consensus state: current_term: 1 leader_uuid: "d3b7224edb394d2f988e69f5794ca85a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } } }
I20260925 15:25:25.381418 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:25.381912 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3b7224edb394d2f988e69f5794ca85a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b7224edb394d2f988e69f5794ca85a" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 35423 } } }
I20260925 15:25:25.381994 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:25.382720 5821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:25.383315 5821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:25.384972 5821 catalog_manager.cc:1347] Generated new cluster ID: 30e02ce3daef47b98849e6d302744086
I20260925 15:25:25.385027 5821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:25.387907 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.376731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.1.163.126:35423)
0925 15:25:25.376920 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:25.376929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:25.376969 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:25:25.377505 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:25.377513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:25.377567 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:25.377734 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:25.378431 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.379326 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.380333 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.381075 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.381552 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.384288 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:25.384292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:25.384292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:25.384305 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:25.386240 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.386679 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.386682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.386682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.386795 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.387021 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.387022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.387023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.387279 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:25.387454 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:25.387568 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:25:25.387714 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260925 15:25:25.406327 5821 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:25.407191 5821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:25.413009 5821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: Generated new TSK 0
I20260925 15:25:25.413240 5821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:25.453142 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:25.574297 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:25.574537 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:25.574564 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:25.575770 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:25.575815 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:25.575845 5832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:25.577458 5832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:25.577715 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:25.577919 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:25.583344 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.583344 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.583410 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:25.584002 5832 server_base.cc:1034] running on GCE node
I20260925 15:25:25.584223 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:25.584506 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:25.585711 5832 hybrid_clock.cc:648] HybridClock initialized: now 1790349925585696 us; error 28 us; skew 500 ppm
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:25.588581 5832 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:25.589027 5832 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:25.589104 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:25.589265 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:25.590426 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "9d30e98c84a146e3894b28d76a55f052"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "4cbe87c123b83e7fb0f98df3b72d5499"
server_key_iv: "031a19769cad03c3e24e1179ac42b86b"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.590806 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "9d30e98c84a146e3894b28d76a55f052"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "4cbe87c123b83e7fb0f98df3b72d5499"
server_key_iv: "031a19769cad03c3e24e1179ac42b86b"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.592191 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:25:25.593115 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.593299 5832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:25.593430 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "9d30e98c84a146e3894b28d76a55f052"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "4cbe87c123b83e7fb0f98df3b72d5499"
server_key_iv: "031a19769cad03c3e24e1179ac42b86b"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.593513 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:25.626778 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:25.629693 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:25.629935 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:25.630242 5832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:25.630606 5832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:25.630661 5832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.630704 5832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:25.630733 5832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.637185 5832 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:46593
I20260925 15:25:25.637215 5959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:46593 every 8 connection(s)
I20260925 15:25:25.637533 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:25.640604 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5832
I20260925 15:25:25.640727 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:25.640928 1677 external_mini_cluster.cc:1428] Setting key 6694adeb099214559ad3a7d99d077eb3
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:25.649011 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.638332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:60015 (local address 127.1.163.126:35423)
0925 15:25:25.638404 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:25.638406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:25.638665 (+ 259us) server_negotiation.cc:407] Connection header received
0925 15:25:25.639197 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:25.639198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:25.639209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:25.639277 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:25.640582 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.641330 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.642473 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.642836 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.642921 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.645750 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:25.645752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:25.645752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:25.645761 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:25.647429 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.647932 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.647934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.647934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.647996 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.648273 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.648275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.648275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.648401 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:25.648479 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:25.648814 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:25:25.648915 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:25:25.649274 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.638311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35423 (local address 127.1.163.65:60015)
0925 15:25:25.638503 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:25.638549 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:25.639080 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:25.639366 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:25.639370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:25.639648 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:25.640418 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.640425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.641432 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.641433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.642370 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.642373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.642931 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.642932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.643105 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.643691 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:25.643703 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:25.645629 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:25.647549 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.647558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.647560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.647832 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.648074 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.648074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.648075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.648197 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.648469 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:25.648475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:25.648673 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:25:25.648794 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:25:25.648970 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:25.649816 5960 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35423
I20260925 15:25:25.649914 5960 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:25.650111 5960 heartbeater.cc:511] Master 127.1.163.126:35423 requested a full tablet report, sending...
I20260925 15:25:25.650668 5770 ts_manager.cc:194] Registered new tserver with Master: 9d30e98c84a146e3894b28d76a55f052 (127.1.163.65:46593)
I20260925 15:25:25.651330 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:60015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:25.673563 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:25.759029 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:25.759251 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:25.759277 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:25.760450 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:25.760490 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:25.760519 5965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:25.762121 5965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:25.762321 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:25.762519 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:25.768020 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.768117 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.768067 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:25.768640 5965 server_base.cc:1034] running on GCE node
I20260925 15:25:25.768872 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:25.769186 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:25.770344 5965 hybrid_clock.cc:648] HybridClock initialized: now 1790349925770326 us; error 39 us; skew 500 ppm
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:25.772948 5965 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:25.773257 5965 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:25.773299 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:25.773402 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:25.774456 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "23b8f0429d294fc680feae4e237df6d7"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7e17974dfe1234e681ba22ca9e0edb20"
server_key_iv: "c720d6d7f7707b3721583df11ab16bc4"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.774765 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "23b8f0429d294fc680feae4e237df6d7"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7e17974dfe1234e681ba22ca9e0edb20"
server_key_iv: "c720d6d7f7707b3721583df11ab16bc4"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.776060 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:25.776813 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.777031 5965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:25.777145 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "23b8f0429d294fc680feae4e237df6d7"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7e17974dfe1234e681ba22ca9e0edb20"
server_key_iv: "c720d6d7f7707b3721583df11ab16bc4"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.777226 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:25.801700 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:25.804373 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:25.804575 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:25.804849 5965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:25.805182 5965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:25.805240 5965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.805282 5965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:25.805335 5965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.811364 5965 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39831
I20260925 15:25:25.811456 6092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39831 every 8 connection(s)
I20260925 15:25:25.811748 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:25.819561 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 5965
I20260925 15:25:25.819697 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:25.819970 1677 external_mini_cluster.cc:1428] Setting key 543dbd67d4381eccab9008e0b424f10a
I20260925 15:25:25.822896 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.812974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:42851 (local address 127.1.163.126:35423)
0925 15:25:25.813067 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:25.813070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:25.813381 (+ 311us) server_negotiation.cc:407] Connection header received
0925 15:25:25.813938 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:25.813939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:25.813952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:25.814013 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:25.815015 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.815531 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.816647 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.816862 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.816962 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.819638 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:25.819641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:25.819641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:25.819652 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:25.821225 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.821741 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.821742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.821742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.821858 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.822170 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.822171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.822171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.822327 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:25.822406 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:25.822658 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:25:25.822756 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:25:25.822918 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.812938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35423 (local address 127.1.163.66:42851)
0925 15:25:25.813194 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:25.813241 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:25.813818 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:25.814102 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:25.814105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:25.814394 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:25.814903 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.814908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.815647 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.815648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.816478 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.816480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.816975 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.816976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.817157 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.817831 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:25.817840 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:25.819533 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:25.821318 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.821325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.821328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.821661 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.821940 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.821942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.821942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.822093 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.822400 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:25.822404 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:25.822529 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 15:25:25.822615 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:25:25.822747 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":141,"threads_started":1}
I20260925 15:25:25.823427 6093 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35423
I20260925 15:25:25.823534 6093 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:25.823696 6093 heartbeater.cc:511] Master 127.1.163.126:35423 requested a full tablet report, sending...
I20260925 15:25:25.824084 5770 ts_manager.cc:194] Registered new tserver with Master: 23b8f0429d294fc680feae4e237df6d7 (127.1.163.66:39831)
I20260925 15:25:25.824484 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:42851
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:25.848618 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:25.932053 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:25.932291 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:25.932317 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:25.933617 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:25.933667 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:25.933698 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:25.935259 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:35423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:25.935516 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:25.935756 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:25.941434 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.941484 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:25.941434 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:25.941803 6098 server_base.cc:1034] running on GCE node
I20260925 15:25:25.942062 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:25.942310 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:25.943468 6098 hybrid_clock.cc:648] HybridClock initialized: now 1790349925943456 us; error 33 us; skew 500 ppm
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:25.946135 6098 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:25.946549 6098 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:25.946602 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:25.946738 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:25.947887 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "e3825fa6d669477dac481c1e043c0041"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7a84f676eeccc73fde09e48983ba4ab8"
server_key_iv: "ee6941569d9f6bf037ce63f6860befc7"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.948267 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "e3825fa6d669477dac481c1e043c0041"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7a84f676eeccc73fde09e48983ba4ab8"
server_key_iv: "ee6941569d9f6bf037ce63f6860befc7"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.949692 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:25.950489 6112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.950858 6098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:25.950953 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "e3825fa6d669477dac481c1e043c0041"
format_stamp: "Formatted at 2026-09-25 15:25:25 on dist-test-slave-hvv5"
server_key: "7a84f676eeccc73fde09e48983ba4ab8"
server_key_iv: "ee6941569d9f6bf037ce63f6860befc7"
server_key_version: "encryptionkey@0"
I20260925 15:25:25.951031 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:25.975699 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:25.978240 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:25.978452 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:25.978753 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:25.979120 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:25.979158 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.979184 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:25.979198 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:25.986128 6098 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:45059
I20260925 15:25:25.986166 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:45059 every 8 connection(s)
I20260925 15:25:25.986516 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:25 dist-test-slave-hvv5 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:25.993993 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6098
I20260925 15:25:25.994096 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:25.994288 1677 external_mini_cluster.cc:1428] Setting key 50aedc5cc4e6ed15f423cea3a9906092
I20260925 15:25:25.996735 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.987314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:47429 (local address 127.1.163.126:35423)
0925 15:25:25.987401 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:25.987404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:25.987657 (+ 253us) server_negotiation.cc:407] Connection header received
0925 15:25:25.988168 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:25.988169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:25.988181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:25.988241 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:25.989338 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.989935 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.990974 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.991159 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.991216 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.993490 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:25.993492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:25.993492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:25.993500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:25.995182 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.995670 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.995671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.995672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.995733 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:25.996028 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:25.996029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:25.996029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:25.996151 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:25.996231 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:25.996556 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:25:25.996643 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:25:25.996963 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:25.987263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:35423 (local address 127.1.163.67:47429)
0925 15:25:25.987518 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:25.987557 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:25.988059 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:25.988308 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:25.988311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:25.988671 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:25.989203 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.989208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.990069 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.990070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.990885 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:25.990887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:25.991235 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:25.991236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:25.991335 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:25.991897 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:25.991905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:25.993389 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:25.995260 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.995267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.995269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.995570 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.995815 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:25.995816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:25.995817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:25.995946 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:25.996222 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:25.996228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:25.996393 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:25:25.996503 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:25:25.996710 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260925 15:25:25.997417 6226 heartbeater.cc:347] Connected to a master server at 127.1.163.126:35423
I20260925 15:25:25.997514 6226 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:25.997646 6226 heartbeater.cc:511] Master 127.1.163.126:35423 requested a full tablet report, sending...
I20260925 15:25:25.998018 5770 ts_manager.cc:194] Registered new tserver with Master: e3825fa6d669477dac481c1e043c0041 (127.1.163.67:45059)
I20260925 15:25:25.998427 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:47429
I20260925 15:25:26.006428 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:26.014873 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.008050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.1.163.126:35423)
0925 15:25:26.008103 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.008106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.008134 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:25:26.008198 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.008199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.008208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.008262 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.009166 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.009660 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.010431 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.010615 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.010673 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.012001 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.012002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.012002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:26.012008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.013627 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:26.014019 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:26.014020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:26.014021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:26.014081 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:26.014277 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:26.014278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:26.014278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:26.014436 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.014518 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.014682 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.014758 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:26.016950 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:26.018702 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:26.025754 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.1.163.65:46593)
0925 15:25:26.022071 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.022074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.022395 (+ 321us) server_negotiation.cc:407] Connection header received
0925 15:25:26.022469 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.022471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.022485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.022560 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:26.022666 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023668 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.024852 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025357 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.025445 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.025508 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.025629 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.025666 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260925 15:25:26.025743 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:46593 (local address 127.0.0.1:51830)
0925 15:25:26.022335 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.022346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.022401 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.022571 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.022572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.022575 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:26.022608 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.022609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023767 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.023768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.024751 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.024754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025446 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.025448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.025523 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.025537 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.025609 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.025663 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":68,"threads_started":1}
I20260925 15:25:26.026029 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:45059 (local address 127.0.0.1:58594)
0925 15:25:26.021643 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.021686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.021752 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.022174 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.022177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.022180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:26.022259 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.022261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023577 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.023578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.024769 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.024771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025717 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.025718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.025789 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.025807 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.025894 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.025948 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":14,"thread_start_us":63,"threads_started":1}
I20260925 15:25:26.026201 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39831 (local address 127.0.0.1:51152)
0925 15:25:26.021837 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.021845 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.021881 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.022271 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.022272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.022275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:26.022347 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.022349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023382 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.023383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.024770 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.024771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025716 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.025717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.025789 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.025951 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.025981 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.026015 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260925 15:25:26.026489 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.1.163.66:39831)
0925 15:25:26.022080 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.022085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.022096 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:26.022136 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.022137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.022153 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.022428 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:26.022499 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023265 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.024874 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025611 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.026069 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.026221 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.026349 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.026393 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":106,"threads_started":1}
I20260925 15:25:26.026607 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.021808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.1.163.67:45059)
0925 15:25:26.022014 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.022019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.022030 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:26.022072 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.022074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.022091 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.022167 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:26.022340 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.023481 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.024874 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.025611 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.026333 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.026391 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.026414 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.026453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260925 15:25:26.026539 5894 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 (DEFAULT_TABLE table=test-table [id=2ec93f9ff451457ba31bddf2b708971a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.026935 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0a98f2166f48949c0d524cfb99b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.027204 6027 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 (DEFAULT_TABLE table=test-table [id=2ec93f9ff451457ba31bddf2b708971a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.027503 6027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0a98f2166f48949c0d524cfb99b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.028941 6160 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 (DEFAULT_TABLE table=test-table [id=2ec93f9ff451457ba31bddf2b708971a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.029249 6160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0a98f2166f48949c0d524cfb99b5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.030422 6243 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: Bootstrap starting.
I20260925 15:25:26.030755 6244 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Bootstrap starting.
I20260925 15:25:26.031148 6243 tablet_bootstrap.cc:654] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.031333 6244 tablet_bootstrap.cc:654] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.031747 6243 log.cc:824] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.031806 6244 log.cc:824] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.032280 6247 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Bootstrap starting.
I20260925 15:25:26.032568 6243 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: No bootstrap required, opened a new log
I20260925 15:25:26.032642 6243 ts_tablet_manager.cc:1397] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:26.032793 6244 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: No bootstrap required, opened a new log
I20260925 15:25:26.032845 6244 ts_tablet_manager.cc:1397] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:26.034648 6243 raft_consensus.cc:348] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.034827 6243 raft_consensus.cc:374] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.034762 6244 raft_consensus.cc:348] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.034862 6243 raft_consensus.cc:729] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d30e98c84a146e3894b28d76a55f052, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.034897 6244 raft_consensus.cc:374] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.034926 6244 raft_consensus.cc:729] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b8f0429d294fc680feae4e237df6d7, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.034989 6243 consensus_queue.cc:260] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.035022 6244 consensus_queue.cc:260] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.035280 6243 ts_tablet_manager.cc:1428] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:26.035280 6244 ts_tablet_manager.cc:1428] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:25:26.035346 5960 heartbeater.cc:503] Master 127.1.163.126:35423 was elected leader, sending a full tablet report...
I20260925 15:25:26.035396 6093 heartbeater.cc:503] Master 127.1.163.126:35423 was elected leader, sending a full tablet report...
I20260925 15:25:26.035945 6247 tablet_bootstrap.cc:654] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.036644 6247 log.cc:824] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.037619 6247 tablet_bootstrap.cc:492] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: No bootstrap required, opened a new log
I20260925 15:25:26.037705 6247 ts_tablet_manager.cc:1397] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:25:26.039189 6247 raft_consensus.cc:348] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.039363 6247 raft_consensus.cc:374] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.039409 6247 raft_consensus.cc:729] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3825fa6d669477dac481c1e043c0041, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.039542 6247 consensus_queue.cc:260] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.039812 6247 ts_tablet_manager.cc:1428] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:26.039849 6226 heartbeater.cc:503] Master 127.1.163.126:35423 was elected leader, sending a full tablet report...
W20260925 15:25:26.062332 6094 tablet.cc:2369] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:26.063436 6248 raft_consensus.cc:482] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:26.063539 6248 raft_consensus.cc:504] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.063983 6248 leader_election.cc:288] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b8f0429d294fc680feae4e237df6d7 (127.1.163.66:39831), e3825fa6d669477dac481c1e043c0041 (127.1.163.67:45059)
I20260925 15:25:26.066811 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.064280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:39063 (local address 127.1.163.66:39831)
0925 15:25:26.064359 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.064361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.064392 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:25:26.064416 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.064417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.064424 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.064494 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:26.064651 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.065120 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.066075 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.066479 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.066587 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.066596 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.066710 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.066745 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:25:26.066823 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.064172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39831 (local address 127.1.163.65:39063)
0925 15:25:26.064258 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.064273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.064339 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.064502 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.064504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.064506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:26.064580 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.064583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.065208 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.065209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.065980 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.065982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.066565 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.066566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.066641 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.066644 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.066700 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.066741 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:25:26.067184 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0a98f2166f48949c0d524cfb99b5d8" candidate_uuid: "9d30e98c84a146e3894b28d76a55f052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8f0429d294fc680feae4e237df6d7" is_pre_election: true
I20260925 15:25:26.067358 6047 raft_consensus.cc:2454] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d30e98c84a146e3894b28d76a55f052 in term 0.
I20260925 15:25:26.067503 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.064461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:46599 (local address 127.1.163.67:45059)
0925 15:25:26.064516 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.064518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.064566 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:25:26.064689 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.064690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.064700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.064790 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:26.064968 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.065429 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.066438 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.067022 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.067094 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.067101 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.067379 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.067424 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:25:26.067523 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.064270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:45059 (local address 127.1.163.65:46599)
0925 15:25:26.064490 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.064501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.064593 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.064799 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.064800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.064802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:26.064887 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.064889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.065523 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.065524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.066322 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:26.066324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.067155 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.067156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:26.067244 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.067268 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.067364 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.067431 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":135,"threads_started":1}
I20260925 15:25:26.067592 5850 leader_election.cc:302] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b8f0429d294fc680feae4e237df6d7, 9d30e98c84a146e3894b28d76a55f052; no voters:
I20260925 15:25:26.067745 6248 raft_consensus.cc:2781] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:26.067814 6248 raft_consensus.cc:482] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:26.067848 6248 raft_consensus.cc:3037] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.067914 6180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0a98f2166f48949c0d524cfb99b5d8" candidate_uuid: "9d30e98c84a146e3894b28d76a55f052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3825fa6d669477dac481c1e043c0041" is_pre_election: true
I20260925 15:25:26.068076 6180 raft_consensus.cc:2454] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d30e98c84a146e3894b28d76a55f052 in term 0.
I20260925 15:25:26.068830 6248 raft_consensus.cc:504] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.068974 6248 leader_election.cc:288] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [CANDIDATE]: Term 1 election: Requested vote from peers 23b8f0429d294fc680feae4e237df6d7 (127.1.163.66:39831), e3825fa6d669477dac481c1e043c0041 (127.1.163.67:45059)
I20260925 15:25:26.069144 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0a98f2166f48949c0d524cfb99b5d8" candidate_uuid: "9d30e98c84a146e3894b28d76a55f052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8f0429d294fc680feae4e237df6d7"
I20260925 15:25:26.069188 6180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0a98f2166f48949c0d524cfb99b5d8" candidate_uuid: "9d30e98c84a146e3894b28d76a55f052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3825fa6d669477dac481c1e043c0041"
I20260925 15:25:26.069252 6180 raft_consensus.cc:3037] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.069252 6047 raft_consensus.cc:3037] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.069983 6180 raft_consensus.cc:2454] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d30e98c84a146e3894b28d76a55f052 in term 1.
I20260925 15:25:26.069988 6047 raft_consensus.cc:2454] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d30e98c84a146e3894b28d76a55f052 in term 1.
I20260925 15:25:26.070158 5848 leader_election.cc:302] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d30e98c84a146e3894b28d76a55f052, e3825fa6d669477dac481c1e043c0041; no voters:
I20260925 15:25:26.070262 6248 raft_consensus.cc:2781] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:26.070408 6248 raft_consensus.cc:686] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [term 1 LEADER]: Becoming Leader. State: Replica: 9d30e98c84a146e3894b28d76a55f052, State: Running, Role: LEADER
I20260925 15:25:26.070504 6248 consensus_queue.cc:237] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } }
I20260925 15:25:26.071557 5770 catalog_manager.cc:5495] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d30e98c84a146e3894b28d76a55f052 (127.1.163.65). New cstate: current_term: 1 leader_uuid: "9d30e98c84a146e3894b28d76a55f052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d30e98c84a146e3894b28d76a55f052" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46593 } health_report { overall_health: HEALTHY } } }
I20260925 15:25:26.084872 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.082039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.1.163.65:46593)
0925 15:25:26.082083 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.082085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.082093 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:26.082178 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.082179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.082195 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.082255 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:26.082440 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.082993 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.083812 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:26.084221 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:26.084285 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:26.084449 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:26.084618 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:26.084695 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.084738 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.084758 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:26.086671 6047 raft_consensus.cc:1261] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 1 FOLLOWER]: Refusing update from remote peer 9d30e98c84a146e3894b28d76a55f052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:26.086709 6180 raft_consensus.cc:1261] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 1 FOLLOWER]: Refusing update from remote peer 9d30e98c84a146e3894b28d76a55f052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:26.087085 6253 consensus_queue.cc:1035] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3825fa6d669477dac481c1e043c0041" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:26.087214 6248 consensus_queue.cc:1035] T 0a0a98f2166f48949c0d524cfb99b5d8 P 9d30e98c84a146e3894b28d76a55f052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b8f0429d294fc680feae4e237df6d7" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:26.090039 6257 mvcc.cc:205] Tried to move back new op lower bound from 7333273297248251904 to 7333273297185972224. Current Snapshot: MvccSnapshot[applied={T|T < 7333273297248251904}]
I20260925 15:25:26.092000 5769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:26.092087 5769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:26.092923 5769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3b7224edb394d2f988e69f5794ca85a: Sending DeleteTablet for 3 replicas of tablet 0a0a98f2166f48949c0d524cfb99b5d8
I20260925 15:25:26.093329 6027 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51152
I20260925 15:25:26.093408 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51830
I20260925 15:25:26.093430 6160 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0a98f2166f48949c0d524cfb99b5d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:58594
I20260925 15:25:26.093647 6268 tablet_replica.cc:330] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: stopping tablet replica
I20260925 15:25:26.093751 6268 raft_consensus.cc:2229] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:26.093794 6266 tablet_replica.cc:330] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: stopping tablet replica
I20260925 15:25:26.093892 6266 raft_consensus.cc:2229] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:26.093904 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5832
I20260925 15:25:26.093986 6266 raft_consensus.cc:2258] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:26.095134 6266 ts_tablet_manager.cc:1905] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:26.093860 6268 raft_consensus.cc:2258] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:26.097004 6268 ts_tablet_manager.cc:1905] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:26.097249 6266 ts_tablet_manager.cc:1918] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:26.097337 6266 log.cc:1196] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/0a0a98f2166f48949c0d524cfb99b5d8
I20260925 15:25:26.097615 6266 ts_tablet_manager.cc:1939] T 0a0a98f2166f48949c0d524cfb99b5d8 P 23b8f0429d294fc680feae4e237df6d7: Deleting consensus metadata
I20260925 15:25:26.098359 5757 catalog_manager.cc:4841] TS 23b8f0429d294fc680feae4e237df6d7 (127.1.163.66:39831): tablet 0a0a98f2166f48949c0d524cfb99b5d8 (table test-table [id=2ec93f9ff451457ba31bddf2b708971a]) successfully deleted
I20260925 15:25:26.098594 6268 ts_tablet_manager.cc:1918] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:26.098664 6268 log.cc:1196] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/0a0a98f2166f48949c0d524cfb99b5d8
I20260925 15:25:26.098955 6268 ts_tablet_manager.cc:1939] T 0a0a98f2166f48949c0d524cfb99b5d8 P e3825fa6d669477dac481c1e043c0041: Deleting consensus metadata
I20260925 15:25:26.099504 5755 catalog_manager.cc:4841] TS e3825fa6d669477dac481c1e043c0041 (127.1.163.67:45059): tablet 0a0a98f2166f48949c0d524cfb99b5d8 (table test-table [id=2ec93f9ff451457ba31bddf2b708971a]) successfully deleted
W20260925 15:25:26.099694 5757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:46593 (error 108)
I20260925 15:25:26.099960 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5965
I20260925 15:25:26.100760 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.100610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:46593 (local address 127.0.0.1:51852)
0925 15:25:26.100668 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.100708 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:25:26.100899 5757 catalog_manager.cc:4584] TS 9d30e98c84a146e3894b28d76a55f052 (127.1.163.65:46593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a0a98f2166f48949c0d524cfb99b5d8: Network error: Client connection negotiation failed: client connection to 127.1.163.65:46593: connect: Connection refused (error 111)
I20260925 15:25:26.104710 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6098
I20260925 15:25:26.109833 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 5739
2026-09-25T15:25:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2076 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:26Z Disabled control of system clock
I20260925 15:25:26.138346 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41109
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:25:26.222550 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:26.222770 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:26.222793 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:26.223922 6273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:26.223959 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:26.223973 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:26.223984 6273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:26.223996 6273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:26.225351 6273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:41109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:26.225562 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:26.225807 6273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:26.230892 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.230908 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.230908 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:26.231236 6273 server_base.cc:1034] running on GCE node
I20260925 15:25:26.231518 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:26.231818 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:26.232990 6273 hybrid_clock.cc:648] HybridClock initialized: now 1790349926232965 us; error 60 us; skew 500 ppm
I20260925 15:25:26.234326 6273 webserver.cc:460] Webserver started at http://127.1.163.126:46149/ using document root <none> and password file <none>
I20260925 15:25:26.234611 6273 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:26.234683 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:26.234835 6273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:26.236106 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "d6d58dca7f0a42cdb1003293aa910402"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "00377d197e87feeab2878a6afd3b296f"
server_key_iv: "a0337b7622a03d9930e9c8b46c8e146d"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.236505 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "d6d58dca7f0a42cdb1003293aa910402"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "00377d197e87feeab2878a6afd3b296f"
server_key_iv: "a0337b7622a03d9930e9c8b46c8e146d"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.237988 6273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:26.238780 6287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.239001 6273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:26.239116 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "d6d58dca7f0a42cdb1003293aa910402"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "00377d197e87feeab2878a6afd3b296f"
server_key_iv: "a0337b7622a03d9930e9c8b46c8e146d"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.239203 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:26.242944 6273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:26.243252 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:26.243414 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:26.247524 6273 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:41109
I20260925 15:25:26.247572 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:41109 every 8 connection(s)
I20260925 15:25:26.247938 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:26.248721 6340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.251554 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: Bootstrap starting.
I20260925 15:25:26.252171 6340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.252530 6340 log.cc:824] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.252653 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6273
I20260925 15:25:26.252753 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:26.252933 1677 external_mini_cluster.cc:1428] Setting key 2a1d573354add4c098ada040d7110345
I20260925 15:25:26.253654 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: No bootstrap required, opened a new log
I20260925 15:25:26.255376 6340 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } }
I20260925 15:25:26.255546 6340 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.255592 6340 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d58dca7f0a42cdb1003293aa910402, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.255719 6340 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } }
I20260925 15:25:26.255818 6340 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:26.255860 6340 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:26.255905 6340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.256494 6340 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } }
I20260925 15:25:26.256645 6340 leader_election.cc:302] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d58dca7f0a42cdb1003293aa910402; no voters:
I20260925 15:25:26.256858 6340 leader_election.cc:288] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:26.257141 6345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:26.257248 6340 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:26.257294 6345 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [term 1 LEADER]: Becoming Leader. State: Replica: d6d58dca7f0a42cdb1003293aa910402, State: Running, Role: LEADER
I20260925 15:25:26.257405 6345 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } }
I20260925 15:25:26.257751 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.253566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.1.163.126:41109)
0925 15:25:26.253824 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.253832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.253857 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:25:26.255396 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.255404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.255464 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.255647 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.256514 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.256517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.256517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.256520 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.256610 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.256627 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.256689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.256792 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.256860 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":115,"threads_started":1}
I20260925 15:25:26.257800 6345 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d58dca7f0a42cdb1003293aa910402. Latest consensus state: current_term: 1 leader_uuid: "d6d58dca7f0a42cdb1003293aa910402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } } }
I20260925 15:25:26.257880 6345 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:26.257803 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d58dca7f0a42cdb1003293aa910402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d58dca7f0a42cdb1003293aa910402" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41109 } } }
I20260925 15:25:26.258133 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:26.258576 6353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:26.259133 6353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:26.260936 6353 catalog_manager.cc:1347] Generated new cluster ID: 78d1a4b831ba4719807d34734eedb4c0
I20260925 15:25:26.260995 6353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:26.289335 6353 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:26.289570 6353 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:26.296092 6353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: Generated new TSK 0
I20260925 15:25:26.296345 6353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:26.308738 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--builtin_ntp_servers=127.1.163.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:25:26.404279 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:26.404495 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:26.404520 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:26.405829 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:26.405877 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:26.405905 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:26.407326 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:26.407605 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:26.407922 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:26.413287 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.413343 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.413311 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:26.413390 6364 server_base.cc:1034] running on GCE node
I20260925 15:25:26.413703 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:26.413955 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:26.415120 6364 hybrid_clock.cc:648] HybridClock initialized: now 1790349926415103 us; error 29 us; skew 500 ppm
I20260925 15:25:26.416507 6364 webserver.cc:460] Webserver started at http://127.1.163.65:34935/ using document root <none> and password file <none>
I20260925 15:25:26.416834 6364 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:26.416893 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:26.416987 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:26.418042 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ec8da5e1544509c46844eafa4b58f103"
server_key_iv: "399cc2767798c63895fb90b9bdb4855a"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.418350 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ec8da5e1544509c46844eafa4b58f103"
server_key_iv: "399cc2767798c63895fb90b9bdb4855a"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.419621 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:26.420364 6378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.420578 6364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:26.420684 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ec8da5e1544509c46844eafa4b58f103"
server_key_iv: "399cc2767798c63895fb90b9bdb4855a"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.420768 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:26.424151 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:26.424418 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:26.424590 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:26.424830 6364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:26.425140 6364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:26.425194 6364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.425233 6364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:26.425262 6364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.431870 6364 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:46075
I20260925 15:25:26.431941 6491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:46075 every 8 connection(s)
I20260925 15:25:26.432284 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:26.434383 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6364
I20260925 15:25:26.434507 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:26.434700 1677 external_mini_cluster.cc:1428] Setting key c6a78fcb7e6f23ee426ec0d06172db29
I20260925 15:25:26.435755 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--builtin_ntp_servers=127.1.163.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:25:26.435869 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.433082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:53037 (local address 127.1.163.126:41109)
0925 15:25:26.433255 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.433258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.433463 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:25:26.434037 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.434038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.434047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.434111 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.435612 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.435613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.435613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.435618 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.435635 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.435640 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.435694 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.435777 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.435802 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260925 15:25:26.436069 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.433074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41109 (local address 127.1.163.65:53037)
0925 15:25:26.433320 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.433373 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.433920 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.434207 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.434211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.434226 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.435448 (+ 1222us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.435448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.435477 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.435506 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.435701 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.435706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.435709 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.435777 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.435841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260925 15:25:26.436734 6492 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41109
I20260925 15:25:26.436842 6492 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:26.436986 6492 heartbeater.cc:511] Master 127.1.163.126:41109 requested a full tablet report, sending...
I20260925 15:25:26.437582 6304 ts_manager.cc:194] Registered new tserver with Master: 5618f21f6e8e4e6eaf5e2fe7170c9caa (127.1.163.65:46075)
W20260925 15:25:26.517082 6495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:26.517309 6495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:26.517331 6495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:26.518553 6495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:26.518599 6495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:26.518627 6495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:26.520005 6495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:26.520270 6495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:26.520612 6495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:25:26.526186 6495 server_base.cc:1034] running on GCE node
W20260925 15:25:26.526186 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.526515 6503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.526580 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:26.526852 6495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:26.527122 6495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:26.528299 6495 hybrid_clock.cc:648] HybridClock initialized: now 1790349926528290 us; error 44 us; skew 500 ppm
I20260925 15:25:26.529735 6495 webserver.cc:460] Webserver started at http://127.1.163.66:38791/ using document root <none> and password file <none>
I20260925 15:25:26.529999 6495 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:26.530071 6495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:26.530224 6495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:26.531417 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "851605e88de94ec1ba39720ea49ae060"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "2c701d464f7a19b9d7d357bce4f752aa"
server_key_iv: "5edb472b85bc3f9cda1054e90ff6fa90"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.531792 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "851605e88de94ec1ba39720ea49ae060"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "2c701d464f7a19b9d7d357bce4f752aa"
server_key_iv: "5edb472b85bc3f9cda1054e90ff6fa90"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.533205 6495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 15:25:26.534014 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.534195 6495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:26.534301 6495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "851605e88de94ec1ba39720ea49ae060"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "2c701d464f7a19b9d7d357bce4f752aa"
server_key_iv: "5edb472b85bc3f9cda1054e90ff6fa90"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.534382 6495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:26.537887 6495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:26.538165 6495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:26.538307 6495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:26.538511 6495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:26.538818 6495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:26.538879 6495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.538923 6495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:26.538950 6495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.545047 6495 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:33417
I20260925 15:25:26.545104 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:33417 every 8 connection(s)
I20260925 15:25:26.545433 6495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:26.548652 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.546239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:39091 (local address 127.1.163.126:41109)
0925 15:25:26.546286 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.546288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.546556 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:25:26.547021 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.547022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.547029 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.547097 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.548371 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.548373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.548373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.548377 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.548393 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.548398 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.548460 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.548518 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.548530 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:25:26.548851 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.546196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41109 (local address 127.1.163.66:39091)
0925 15:25:26.546381 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.546424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.546911 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.547167 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.547171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.547179 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.548214 (+ 1035us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.548215 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.548243 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.548272 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.548459 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.548461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.548464 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.548516 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.548631 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260925 15:25:26.549329 6623 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41109
I20260925 15:25:26.549407 6623 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:26.549592 6623 heartbeater.cc:511] Master 127.1.163.126:41109 requested a full tablet report, sending...
I20260925 15:25:26.550088 6304 ts_manager.cc:194] Registered new tserver with Master: 851605e88de94ec1ba39720ea49ae060 (127.1.163.66:33417)
I20260925 15:25:26.551210 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6495
I20260925 15:25:26.551285 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:26.551517 1677 external_mini_cluster.cc:1428] Setting key 065a376c65503393fdf97d96cedd7880
I20260925 15:25:26.552428 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--builtin_ntp_servers=127.1.163.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:25:26.635358 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:26.635588 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:26.635612 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:26.636790 6626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:26.636825 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:26.636852 6626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:26.638334 6626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:26.638535 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:26.638775 6626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:26.643909 6632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.643903 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.643931 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:26.644027 6626 server_base.cc:1034] running on GCE node
I20260925 15:25:26.644280 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:26.644492 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:26.645715 6626 hybrid_clock.cc:648] HybridClock initialized: now 1790349926645702 us; error 31 us; skew 500 ppm
I20260925 15:25:26.647032 6626 webserver.cc:460] Webserver started at http://127.1.163.67:40833/ using document root <none> and password file <none>
I20260925 15:25:26.647284 6626 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:26.647354 6626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:26.647514 6626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:26.648674 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "b7ba1b038e4a4a9a94212039b94dd9c4"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ae537c317d0c4f85433bc206be81ecf6"
server_key_iv: "a5d7919c85fe77d2ebff900b9b9fb763"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.649052 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "b7ba1b038e4a4a9a94212039b94dd9c4"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ae537c317d0c4f85433bc206be81ecf6"
server_key_iv: "a5d7919c85fe77d2ebff900b9b9fb763"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.650384 6626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:26.651129 6640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.651297 6626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:26.651409 6626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b7ba1b038e4a4a9a94212039b94dd9c4"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "ae537c317d0c4f85433bc206be81ecf6"
server_key_iv: "a5d7919c85fe77d2ebff900b9b9fb763"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.651489 6626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:26.655027 6626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:26.655351 6626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:26.655507 6626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:26.655753 6626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:26.656136 6626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:26.656174 6626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.656196 6626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:26.656208 6626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.662230 6626 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:38751
I20260925 15:25:26.662272 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:38751 every 8 connection(s)
I20260925 15:25:26.662636 6626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:26.666023 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.663494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:39633 (local address 127.1.163.126:41109)
0925 15:25:26.663563 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.663566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.663815 (+ 249us) server_negotiation.cc:407] Connection header received
0925 15:25:26.664268 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.664268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.664274 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.664354 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.665684 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.665685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.665685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.665690 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.665707 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.665712 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.665776 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.665946 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.665962 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:25:26.666200 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.663486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41109 (local address 127.1.163.67:39633)
0925 15:25:26.663688 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.663723 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.664181 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.664458 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.664461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.664470 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.665506 (+ 1036us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.665507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.665537 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.665567 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.665822 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.665829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.665832 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.665911 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.665966 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260925 15:25:26.666910 6754 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41109
I20260925 15:25:26.667047 6754 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:26.667232 6754 heartbeater.cc:511] Master 127.1.163.126:41109 requested a full tablet report, sending...
I20260925 15:25:26.667364 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6626
I20260925 15:25:26.667471 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:26.667680 1677 external_mini_cluster.cc:1428] Setting key 8479561b572665af6911e82c94abc6dc
I20260925 15:25:26.667672 6304 ts_manager.cc:194] Registered new tserver with Master: b7ba1b038e4a4a9a94212039b94dd9c4 (127.1.163.67:38751)
I20260925 15:25:26.668448 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:26.670852 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.669932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.1.163.126:41109)
0925 15:25:26.669989 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.669991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.670026 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:25:26.670088 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.670088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.670093 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.670150 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.670677 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.670678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.670679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.670683 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.670699 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.670703 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.670756 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.670789 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.670802 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:25:26.672653 6304 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:26.674181 6304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:26.678462 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38751 (local address 127.0.0.1:38188)
0925 15:25:26.676803 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.676816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.676871 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.677407 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.677408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.677411 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.677947 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.677947 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.677962 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.678007 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.678270 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.678271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.678272 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.678327 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.678342 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260925 15:25:26.678632 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.1.163.65:46075)
0925 15:25:26.677138 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.677142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.677151 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:26.677176 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.677178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.677186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.677546 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.678122 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.678130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.678130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.678134 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.678162 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.678186 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.678246 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.678522 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.678541 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":82,"threads_started":1}
I20260925 15:25:26.678658 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.1.163.66:33417)
0925 15:25:26.677109 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.677114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.677144 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:25:26.677172 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.677174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.677184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.677294 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.678302 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.678309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.678309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.678314 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.678344 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.678369 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.678430 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.678558 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.678579 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260925 15:25:26.678932 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.1.163.67:38751)
0925 15:25:26.676946 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.676951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.676962 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:26.676991 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.676993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.677004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.677083 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.678110 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.678118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.678119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.678123 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.678160 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.678186 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.678244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.678847 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.678871 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":110,"threads_started":1}
I20260925 15:25:26.678478 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:46075 (local address 127.0.0.1:37722)
0925 15:25:26.676587 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.676630 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.676689 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.677310 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.677316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.677323 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.678017 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.678018 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.678023 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.678032 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.678246 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.678254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.678257 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.678333 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.678347 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260925 15:25:26.678575 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.676450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:33417 (local address 127.0.0.1:45836)
0925 15:25:26.676960 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.676969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.677005 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.677605 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.677606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.677610 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.678195 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.678196 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.678204 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.678227 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.678439 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.678440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.678440 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.678509 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.678522 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":2,"thread_start_us":224,"threads_started":1}
I20260925 15:25:26.679776 6688 tablet_service.cc:1467] Processing CreateTablet for tablet e5eed82c53db4efc90c9c6aebca11968 (DEFAULT_TABLE table=test-table [id=8905d716f8704439b20cecf1557fc197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.679824 6426 tablet_service.cc:1467] Processing CreateTablet for tablet e5eed82c53db4efc90c9c6aebca11968 (DEFAULT_TABLE table=test-table [id=8905d716f8704439b20cecf1557fc197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.680075 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eed82c53db4efc90c9c6aebca11968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.679986 6557 tablet_service.cc:1467] Processing CreateTablet for tablet e5eed82c53db4efc90c9c6aebca11968 (DEFAULT_TABLE table=test-table [id=8905d716f8704439b20cecf1557fc197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:26.680193 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eed82c53db4efc90c9c6aebca11968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.680289 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eed82c53db4efc90c9c6aebca11968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:26.682652 6768 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Bootstrap starting.
I20260925 15:25:26.683626 6768 tablet_bootstrap.cc:654] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.683643 6770 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Bootstrap starting.
I20260925 15:25:26.683945 6768 log.cc:824] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.684280 6770 tablet_bootstrap.cc:654] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.684628 6770 log.cc:824] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.684734 6769 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: Bootstrap starting.
I20260925 15:25:26.684821 6768 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: No bootstrap required, opened a new log
I20260925 15:25:26.684890 6768 ts_tablet_manager.cc:1397] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:26.685391 6769 tablet_bootstrap.cc:654] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:26.685489 6770 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: No bootstrap required, opened a new log
I20260925 15:25:26.685555 6770 ts_tablet_manager.cc:1397] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:25:26.685739 6769 log.cc:824] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:26.686530 6769 tablet_bootstrap.cc:492] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: No bootstrap required, opened a new log
I20260925 15:25:26.686619 6769 ts_tablet_manager.cc:1397] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:26.687829 6768 raft_consensus.cc:348] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.688057 6768 raft_consensus.cc:374] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.688100 6768 raft_consensus.cc:729] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ba1b038e4a4a9a94212039b94dd9c4, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.688251 6768 consensus_queue.cc:260] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.688387 6770 raft_consensus.cc:348] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.688570 6768 ts_tablet_manager.cc:1428] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:25:26.688570 6770 raft_consensus.cc:374] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.688612 6770 raft_consensus.cc:729] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851605e88de94ec1ba39720ea49ae060, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.688689 6754 heartbeater.cc:503] Master 127.1.163.126:41109 was elected leader, sending a full tablet report...
I20260925 15:25:26.688724 6770 consensus_queue.cc:260] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.688787 6769 raft_consensus.cc:348] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.688982 6770 ts_tablet_manager.cc:1428] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 15:25:26.689157 6623 heartbeater.cc:503] Master 127.1.163.126:41109 was elected leader, sending a full tablet report...
I20260925 15:25:26.689512 6769 raft_consensus.cc:374] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:26.689612 6769 raft_consensus.cc:729] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618f21f6e8e4e6eaf5e2fe7170c9caa, State: Initialized, Role: FOLLOWER
I20260925 15:25:26.689733 6769 consensus_queue.cc:260] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.690014 6769 ts_tablet_manager.cc:1428] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:25:26.690097 6492 heartbeater.cc:503] Master 127.1.163.126:41109 was elected leader, sending a full tablet report...
I20260925 15:25:26.790493 6776 raft_consensus.cc:482] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:26.790644 6776 raft_consensus.cc:504] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.791034 6776 leader_election.cc:288] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7ba1b038e4a4a9a94212039b94dd9c4 (127.1.163.67:38751), 851605e88de94ec1ba39720ea49ae060 (127.1.163.66:33417)
I20260925 15:25:26.792775 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.791281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38751 (local address 127.1.163.65:49159)
0925 15:25:26.791361 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.791384 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.791447 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.791803 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.791804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.791807 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.792349 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.792349 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.792358 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.792375 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.792622 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.792624 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.792624 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.792680 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.792695 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:25:26.792860 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.791345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:49159 (local address 127.1.163.67:38751)
0925 15:25:26.791393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.791395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.791463 (+ 68us) server_negotiation.cc:407] Connection header received
0925 15:25:26.791511 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.791530 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.791538 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.791617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.792476 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.792478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.792478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.792484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.792504 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.792510 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.792622 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.792684 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.792696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:25:26.792899 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.791288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:33417 (local address 127.1.163.65:51099)
0925 15:25:26.791586 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.791597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:26.791651 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:26.791887 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:26.791888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:26.791891 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:26.792529 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:26.792529 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:26.792567 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:26.792579 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:26.792749 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:26.792750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:26.792751 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:25:26.792801 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:25:26.792815 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":117,"thread_start_us":108,"threads_started":1}
I20260925 15:25:26.792897 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.791365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:51099 (local address 127.1.163.66:33417)
0925 15:25:26.791407 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.791409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.791707 (+ 298us) server_negotiation.cc:407] Connection header received
0925 15:25:26.791739 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.791740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.791776 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.791868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.792650 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.792651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.792651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.792657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.792676 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.792681 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.792738 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.792799 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.792813 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:25:26.793193 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eed82c53db4efc90c9c6aebca11968" candidate_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" is_pre_election: true
I20260925 15:25:26.793274 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eed82c53db4efc90c9c6aebca11968" candidate_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851605e88de94ec1ba39720ea49ae060" is_pre_election: true
I20260925 15:25:26.793367 6708 raft_consensus.cc:2454] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618f21f6e8e4e6eaf5e2fe7170c9caa in term 0.
I20260925 15:25:26.793442 6577 raft_consensus.cc:2454] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5618f21f6e8e4e6eaf5e2fe7170c9caa in term 0.
I20260925 15:25:26.793637 6381 leader_election.cc:302] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5618f21f6e8e4e6eaf5e2fe7170c9caa, b7ba1b038e4a4a9a94212039b94dd9c4; no voters:
I20260925 15:25:26.793788 6776 raft_consensus.cc:2781] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:26.793849 6776 raft_consensus.cc:482] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:26.793872 6776 raft_consensus.cc:3037] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.794687 6776 raft_consensus.cc:504] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.794975 6776 leader_election.cc:288] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [CANDIDATE]: Term 1 election: Requested vote from peers b7ba1b038e4a4a9a94212039b94dd9c4 (127.1.163.67:38751), 851605e88de94ec1ba39720ea49ae060 (127.1.163.66:33417)
I20260925 15:25:26.795015 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eed82c53db4efc90c9c6aebca11968" candidate_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4"
I20260925 15:25:26.795112 6708 raft_consensus.cc:3037] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.795993 6708 raft_consensus.cc:2454] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618f21f6e8e4e6eaf5e2fe7170c9caa in term 1.
W20260925 15:25:26.796002 6624 tablet.cc:2369] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:26.796147 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eed82c53db4efc90c9c6aebca11968" candidate_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851605e88de94ec1ba39720ea49ae060"
I20260925 15:25:26.796229 6577 raft_consensus.cc:3037] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:26.796267 6381 leader_election.cc:302] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5618f21f6e8e4e6eaf5e2fe7170c9caa, b7ba1b038e4a4a9a94212039b94dd9c4; no voters:
I20260925 15:25:26.796360 6776 raft_consensus.cc:2781] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:26.796677 6776 raft_consensus.cc:686] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [term 1 LEADER]: Becoming Leader. State: Replica: 5618f21f6e8e4e6eaf5e2fe7170c9caa, State: Running, Role: LEADER
I20260925 15:25:26.796813 6776 consensus_queue.cc:237] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } }
I20260925 15:25:26.797268 6577 raft_consensus.cc:2454] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5618f21f6e8e4e6eaf5e2fe7170c9caa in term 1.
I20260925 15:25:26.797734 6304 catalog_manager.cc:5495] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5618f21f6e8e4e6eaf5e2fe7170c9caa (127.1.163.65). New cstate: current_term: 1 leader_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5618f21f6e8e4e6eaf5e2fe7170c9caa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 46075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:26.827648 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.826458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.1.163.65:46075)
0925 15:25:26.826547 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:26.826549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:26.826557 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:26.826606 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:26.826608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:26.826621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:26.826682 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:26.827410 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:26.827412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:26.827412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:26.827416 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:26.827433 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:26.827437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:26.827495 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:26.827551 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:25:26.827567 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:25:26.829751 6708 raft_consensus.cc:1261] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 1 FOLLOWER]: Refusing update from remote peer 5618f21f6e8e4e6eaf5e2fe7170c9caa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:26.829792 6577 raft_consensus.cc:1261] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 1 FOLLOWER]: Refusing update from remote peer 5618f21f6e8e4e6eaf5e2fe7170c9caa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:26.830189 6776 consensus_queue.cc:1035] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ba1b038e4a4a9a94212039b94dd9c4" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:26.830349 6778 consensus_queue.cc:1035] T e5eed82c53db4efc90c9c6aebca11968 P 5618f21f6e8e4e6eaf5e2fe7170c9caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "851605e88de94ec1ba39720ea49ae060" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 33417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:26.833299 6783 mvcc.cc:205] Tried to move back new op lower bound from 7333273300291530752 to 7333273300160540672. Current Snapshot: MvccSnapshot[applied={T|T < 7333273300291530752}]
I20260925 15:25:26.833936 6781 mvcc.cc:205] Tried to move back new op lower bound from 7333273300291530752 to 7333273300160540672. Current Snapshot: MvccSnapshot[applied={T|T < 7333273300291530752 or (T in {7333273300291530752})}]
I20260925 15:25:26.835557 6304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:26.835660 6304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:26.836504 6304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6d58dca7f0a42cdb1003293aa910402: Sending DeleteTablet for 3 replicas of tablet e5eed82c53db4efc90c9c6aebca11968
I20260925 15:25:26.836915 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eed82c53db4efc90c9c6aebca11968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='slave'} at 127.0.0.1:37722
I20260925 15:25:26.836942 6688 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eed82c53db4efc90c9c6aebca11968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='slave'} at 127.0.0.1:38188
I20260925 15:25:26.837286 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6364
I20260925 15:25:26.837323 6791 tablet_replica.cc:330] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: stopping tablet replica
I20260925 15:25:26.837309 6557 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eed82c53db4efc90c9c6aebca11968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:26 UTC) from {username='slave'} at 127.0.0.1:45836
I20260925 15:25:26.837423 6791 raft_consensus.cc:2229] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:26.837524 6791 raft_consensus.cc:2258] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:26.840852 6793 tablet_replica.cc:330] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: stopping tablet replica
I20260925 15:25:26.840948 6791 ts_tablet_manager.cc:1905] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:26.841008 6793 raft_consensus.cc:2229] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:26.841138 6793 raft_consensus.cc:2258] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:26.841617 6793 ts_tablet_manager.cc:1905] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:25:26.843446 6289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:46075 (error 108)
I20260925 15:25:26.843637 6793 ts_tablet_manager.cc:1918] T e5eed82c53db4efc90c9c6aebca11968 P 851605e88de94ec1ba39720ea49ae060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:26.843678 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6495
I20260925 15:25:26.843710 6791 ts_tablet_manager.cc:1918] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:26.844877 6791 log.cc:1196] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/e5eed82c53db4efc90c9c6aebca11968
I20260925 15:25:26.844933 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.844764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:46075 (local address 127.0.0.1:37746)
0925 15:25:26.844838 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.844892 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:46075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:25:26.845048 6289 catalog_manager.cc:4584] TS 5618f21f6e8e4e6eaf5e2fe7170c9caa (127.1.163.65:46075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5eed82c53db4efc90c9c6aebca11968: Network error: Client connection negotiation failed: client connection to 127.1.163.65:46075: connect: Connection refused (error 111)
I20260925 15:25:26.845335 6791 ts_tablet_manager.cc:1939] T e5eed82c53db4efc90c9c6aebca11968 P b7ba1b038e4a4a9a94212039b94dd9c4: Deleting consensus metadata
I20260925 15:25:26.845781 6290 catalog_manager.cc:4841] TS b7ba1b038e4a4a9a94212039b94dd9c4 (127.1.163.67:38751): tablet e5eed82c53db4efc90c9c6aebca11968 (table test-table [id=8905d716f8704439b20cecf1557fc197]) successfully deleted
I20260925 15:25:26.848999 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6626
I20260925 15:25:26.849041 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:26.848897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:33417 (local address 127.0.0.1:45842)
0925 15:25:26.848960 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:26.848996 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.66:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:25:26.853539 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6273
2026-09-25T15:25:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (742 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:25:26.873709 1677 mini_oidc.cc:152] Starting JWKS server
I20260925 15:25:26.874864 1677 webserver.cc:460] Webserver started at https://127.0.0.1:42327/ using document root <none> and password file <none>
I20260925 15:25:26.874990 1677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:25:26.875180 1677 webserver.cc:460] Webserver started at http://127.0.0.1:38907/ using document root <none> and password file <none>
2026-09-25T15:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:26Z Disabled control of system clock
I20260925 15:25:26.883349 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37239
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37239
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42327/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/testchainca.cert with env {}
W20260925 15:25:26.966763 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:26.967006 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:26.967031 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:26.968300 6800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:26.968345 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:26.968358 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:26.968371 6800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:26.968384 6800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:25:26.968395 6800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:25:26.968451 6800 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42327/jwks.json
I20260925 15:25:26.970388 6800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42327/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:26.970677 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:26.970927 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:26.976617 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.976620 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:26.976617 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:26.976655 6800 server_base.cc:1034] running on GCE node
I20260925 15:25:26.977260 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:26.977568 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:26.978749 6800 hybrid_clock.cc:648] HybridClock initialized: now 1790349926978686 us; error 83 us; skew 500 ppm
I20260925 15:25:26.980188 6800 webserver.cc:460] Webserver started at http://127.1.163.126:43857/ using document root <none> and password file <none>
I20260925 15:25:26.980463 6800 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:26.980540 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:26.980715 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:26.981832 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "53ca4f09b16945e080d31de660dc95d5"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "efe907989cbe53846f1de5bfad6a87b4"
server_key_iv: "3fa226f02938c72b4f9183baa555f4ad"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.982208 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "53ca4f09b16945e080d31de660dc95d5"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "efe907989cbe53846f1de5bfad6a87b4"
server_key_iv: "3fa226f02938c72b4f9183baa555f4ad"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.983651 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:26.984623 6814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:26.984815 6800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:26.984917 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "53ca4f09b16945e080d31de660dc95d5"
format_stamp: "Formatted at 2026-09-25 15:25:26 on dist-test-slave-hvv5"
server_key: "efe907989cbe53846f1de5bfad6a87b4"
server_key_iv: "3fa226f02938c72b4f9183baa555f4ad"
server_key_version: "encryptionkey@0"
I20260925 15:25:26.985006 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:26.995590 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:26.995956 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:26.996119 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:27.000240 6800 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37239
I20260925 15:25:27.000277 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37239 every 8 connection(s)
I20260925 15:25:27.000706 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:27.001848 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:27.004254 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5: Bootstrap starting.
I20260925 15:25:27.004966 6867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:27.005272 6867 log.cc:824] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:27.006060 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5: No bootstrap required, opened a new log
I20260925 15:25:27.007691 6867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } }
I20260925 15:25:27.007876 6867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:27.007925 6867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ca4f09b16945e080d31de660dc95d5, State: Initialized, Role: FOLLOWER
I20260925 15:25:27.008103 6867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } }
I20260925 15:25:27.008205 6867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:27.008183 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6800
I20260925 15:25:27.008250 6867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:27.008285 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:27.008301 6867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:27.008491 1677 external_mini_cluster.cc:1428] Setting key c5c32db2b69479ae4537cf958740ad9e
I20260925 15:25:27.009119 6867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } }
I20260925 15:25:27.009259 6867 leader_election.cc:302] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ca4f09b16945e080d31de660dc95d5; no voters:
I20260925 15:25:27.009442 6867 leader_election.cc:288] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:27.009488 6872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:27.009606 6872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [term 1 LEADER]: Becoming Leader. State: Replica: 53ca4f09b16945e080d31de660dc95d5, State: Running, Role: LEADER
I20260925 15:25:27.009688 6872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } }
I20260925 15:25:27.009720 6867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:27.010031 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ca4f09b16945e080d31de660dc95d5. Latest consensus state: current_term: 1 leader_uuid: "53ca4f09b16945e080d31de660dc95d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } } }
I20260925 15:25:27.010103 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.010267 6874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ca4f09b16945e080d31de660dc95d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ca4f09b16945e080d31de660dc95d5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37239 } } }
I20260925 15:25:27.010423 6874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.010324 6877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:27.011021 6877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:27.012800 6877 catalog_manager.cc:1347] Generated new cluster ID: 540d148607084550801f68c326dc4a82
I20260925 15:25:27.012861 6877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:27.015216 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.009062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.1.163.126:37239)
0925 15:25:27.009344 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.009372 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.009417 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:25:27.009575 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.009583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.009666 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.010115 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:27.010221 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.011741 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.012779 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.013369 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.014256 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.014417 (+ 161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:27.014420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:27.014420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:27.014613 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:27.014648 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:27.014661 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:27.014892 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:27.014924 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:25:27.015011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260925 15:25:27.016106 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:25:27.025928 6877 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:27.026626 6877 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:27.040693 6877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ca4f09b16945e080d31de660dc95d5: Generated new TSK 0
I20260925 15:25:27.040961 6877 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:25:27.071053 6897 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:25:27.071225 6897 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:25:27.071408 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.048480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.1.163.126:37239)
0925 15:25:27.048572 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.048575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.048702 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:25:27.048845 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.048846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.048877 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.048968 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:25:27.049129 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.049660 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.050483 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.050695 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.050783 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.051034 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:25:27.070978 (+ 19944us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:25:27.071332 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:25:27.071475 6871 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40172: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:25:27.071693 6896 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:27.048400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37239 (local address 127.0.0.1:40172)
0925 15:25:27.048616 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:27.048627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:27.048727 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:27.048977 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:27.048978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:27.048980 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:25:27.049047 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.049050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.049750 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.049752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.050377 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.050379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.050788 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.050789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.050865 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.050894 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:25:27.071170 (+ 20276us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:25:27.071317 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.163.126:37239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":191,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.163.126:37239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42327/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:25:27.072116 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6800
2026-09-25T15:25:27Z chronyd exiting
I20260925 15:25:27.108738 1677 test_util.cc:175] -----------------------------------------------
I20260925 15:25:27.108844 1677 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349904034090-1677-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:25:27.109709 1677 mini_oidc.cc:152] Starting JWKS server
I20260925 15:25:27.110431 1677 webserver.cc:460] Webserver started at https://127.0.0.1:39253/ using document root <none> and password file <none>
I20260925 15:25:27.110507 1677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:25:27.110674 1677 webserver.cc:460] Webserver started at http://127.0.0.1:43099/ using document root <none> and password file <none>
2026-09-25T15:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:27Z Disabled control of system clock
I20260925 15:25:27.118940 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41177
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:41177
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/testchainca.cert with env {}
W20260925 15:25:27.201815 6903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:27.202080 6903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:27.202106 6903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:27.203670 6903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:27.203725 6903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:27.203737 6903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:27.203751 6903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:27.203763 6903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:25:27.203776 6903 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:25:27.203838 6903 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39253/jwks.json
I20260925 15:25:27.205528 6903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:41177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:27.205818 6903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:27.206055 6903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:27.211180 6911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:27.211277 6909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:27.211180 6908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:27.211434 6903 server_base.cc:1034] running on GCE node
I20260925 15:25:27.211715 6903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:27.212023 6903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:27.213201 6903 hybrid_clock.cc:648] HybridClock initialized: now 1790349927213187 us; error 46 us; skew 500 ppm
I20260925 15:25:27.214730 6903 webserver.cc:460] Webserver started at http://127.1.163.126:41979/ using document root <none> and password file <none>
I20260925 15:25:27.215008 6903 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:27.215087 6903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:27.215246 6903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:27.216389 6903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "8cc7653c805048daa67f4a8c3ca067cd"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "655a362817fce0082a852c46e4bbb03c"
server_key_iv: "77493c68766c4fdf082189accabeae00"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.216822 6903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "8cc7653c805048daa67f4a8c3ca067cd"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "655a362817fce0082a852c46e4bbb03c"
server_key_iv: "77493c68766c4fdf082189accabeae00"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.218317 6903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:27.219195 6917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:27.219408 6903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:27.219529 6903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "8cc7653c805048daa67f4a8c3ca067cd"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "655a362817fce0082a852c46e4bbb03c"
server_key_iv: "77493c68766c4fdf082189accabeae00"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.219621 6903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:27.236752 6903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:27.237130 6903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:27.237305 6903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:27.241346 6903 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:41177
I20260925 15:25:27.241379 6969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:41177 every 8 connection(s)
I20260925 15:25:27.241770 6903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:27.242805 6970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:27.243626 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 6903
I20260925 15:25:27.243742 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:27.243935 1677 external_mini_cluster.cc:1428] Setting key 4f701c023dd6ca2200af066cce919a16
I20260925 15:25:27.246634 6970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd: Bootstrap starting.
I20260925 15:25:27.247442 6970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:27.247747 6970 log.cc:824] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:27.248608 6970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd: No bootstrap required, opened a new log
I20260925 15:25:27.250927 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.244666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.1.163.126:41177)
0925 15:25:27.244902 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.244913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.244948 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:25:27.245524 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.245533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.245630 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.245807 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:27.246026 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.247463 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.248370 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.249005 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.249137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.250066 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:27.250073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:27.250073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:27.250299 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:27.250344 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:27.250361 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:27.250473 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:27.250553 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:25:27.250661 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 15:25:27.251498 6970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } }
I20260925 15:25:27.251678 6970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:27.251714 6970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc7653c805048daa67f4a8c3ca067cd, State: Initialized, Role: FOLLOWER
I20260925 15:25:27.251861 6970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } }
I20260925 15:25:27.251958 6970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:27.251994 6970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:27.252045 6970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:27.252122 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:25:27.252796 6970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } }
I20260925 15:25:27.252949 6970 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc7653c805048daa67f4a8c3ca067cd; no voters:
I20260925 15:25:27.253170 6970 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:27.253199 6975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:27.253366 6975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [term 1 LEADER]: Becoming Leader. State: Replica: 8cc7653c805048daa67f4a8c3ca067cd, State: Running, Role: LEADER
I20260925 15:25:27.253458 6975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } }
I20260925 15:25:27.253458 6970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:27.253811 6977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc7653c805048daa67f4a8c3ca067cd. Latest consensus state: current_term: 1 leader_uuid: "8cc7653c805048daa67f4a8c3ca067cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } } }
I20260925 15:25:27.253799 6976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc7653c805048daa67f4a8c3ca067cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc7653c805048daa67f4a8c3ca067cd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41177 } } }
I20260925 15:25:27.253898 6977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.253907 6976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.254142 6979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:27.254693 6979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:27.256505 6979 catalog_manager.cc:1347] Generated new cluster ID: 03549dbb9f234d8385507c2c6cfb429b
I20260925 15:25:27.256599 6979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:27.278955 6979 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:27.279707 6979 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:27.291698 6979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc7653c805048daa67f4a8c3ca067cd: Generated new TSK 0
I20260925 15:25:27.292037 6979 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:25:27.331632 7000 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:25:27.331733 7000 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:25:27.332039 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.310008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.1.163.126:41177)
0925 15:25:27.310094 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.310096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.310196 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:25:27.310274 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.310275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.310288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.310361 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:25:27.310532 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.311251 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.312504 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.312866 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.312928 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.313158 (+ 230us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:25:27.331765 (+ 18607us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:25:27.331982 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 15:25:27.332111 6973 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:25:27.332194 6999 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:27.309936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41177 (local address 127.0.0.1:34906)
0925 15:25:27.310104 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:27.310116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:27.310191 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:27.310380 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:27.310380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:27.310382 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:25:27.310441 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.310444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.311372 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.311373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.312390 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.312394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.312954 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.312956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.313021 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.313028 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:25:27.331957 (+ 18929us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:25:27.332096 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.163.126:41177: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260925 15:25:27.332418 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 6903
2026-09-25T15:25:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (261 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:25:27.370901 1677 mini_oidc.cc:152] Starting JWKS server
I20260925 15:25:27.371644 1677 webserver.cc:460] Webserver started at https://127.0.0.1:43981/ using document root <none> and password file <none>
I20260925 15:25:27.371694 1677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:25:27.371860 1677 webserver.cc:460] Webserver started at http://127.0.0.1:37399/ using document root <none> and password file <none>
2026-09-25T15:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:27Z Disabled control of system clock
I20260925 15:25:27.380316 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44669
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44669
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:25:27.463874 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:27.464128 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:27.464154 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:27.465610 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:27.465667 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:27.465680 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:27.465695 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:27.465709 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:25:27.465723 7006 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:25:27.465772 7006 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43981/jwks.json
I20260925 15:25:27.467371 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:27.467690 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:27.468046 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:27.473376 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:27.473336 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:27.473343 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:27.474120 7006 server_base.cc:1034] running on GCE node
I20260925 15:25:27.474390 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:27.474761 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:27.475943 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790349927475994 us; error 95 us; skew 500 ppm
I20260925 15:25:27.477346 7006 webserver.cc:460] Webserver started at http://127.1.163.126:41541/ using document root <none> and password file <none>
I20260925 15:25:27.477598 7006 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:27.477670 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:27.477826 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:27.478987 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "e9c13a0705f646d888fcf2cd5f6a64e5"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "23055947910cfd206a820afc9d592de1"
server_key_iv: "edef1d0e567d0579859ae1f583f9b0f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.479370 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "e9c13a0705f646d888fcf2cd5f6a64e5"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "23055947910cfd206a820afc9d592de1"
server_key_iv: "edef1d0e567d0579859ae1f583f9b0f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.480829 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:27.481684 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:27.481902 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:27.482015 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "e9c13a0705f646d888fcf2cd5f6a64e5"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "23055947910cfd206a820afc9d592de1"
server_key_iv: "edef1d0e567d0579859ae1f583f9b0f3"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.482096 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:27.498716 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:27.499114 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:27.499303 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:27.503737 7006 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44669
I20260925 15:25:27.503778 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44669 every 8 connection(s)
I20260925 15:25:27.504208 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:27.504894 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7006
I20260925 15:25:27.504997 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:27.505119 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:27.505185 1677 external_mini_cluster.cc:1428] Setting key 092f736dbb26d70a40a820d6b77307cb
I20260925 15:25:27.509007 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5: Bootstrap starting.
I20260925 15:25:27.509757 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:27.510102 7073 log.cc:824] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:27.510874 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5: No bootstrap required, opened a new log
I20260925 15:25:27.511082 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.505740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.1.163.126:44669)
0925 15:25:27.505950 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.505957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.505982 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:25:27.506411 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.506418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.506492 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.506609 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:27.506930 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.507807 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.508963 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.509430 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.509528 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.510331 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:27.510336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:27.510336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:27.510522 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:27.510557 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:27.510569 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:27.510623 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:27.510728 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:25:27.510830 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260925 15:25:27.512239 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:25:27.513367 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } }
I20260925 15:25:27.513542 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:27.513592 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c13a0705f646d888fcf2cd5f6a64e5, State: Initialized, Role: FOLLOWER
I20260925 15:25:27.513715 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } }
I20260925 15:25:27.513787 7073 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:27.513842 7073 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:27.513891 7073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:27.514484 7073 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } }
I20260925 15:25:27.514572 7073 leader_election.cc:302] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9c13a0705f646d888fcf2cd5f6a64e5; no voters:
I20260925 15:25:27.514755 7073 leader_election.cc:288] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:27.514882 7078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:27.515017 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:27.515059 7078 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [term 1 LEADER]: Becoming Leader. State: Replica: e9c13a0705f646d888fcf2cd5f6a64e5, State: Running, Role: LEADER
I20260925 15:25:27.515162 7078 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } }
I20260925 15:25:27.515545 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } } }
I20260925 15:25:27.515556 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9c13a0705f646d888fcf2cd5f6a64e5. Latest consensus state: current_term: 1 leader_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c13a0705f646d888fcf2cd5f6a64e5" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44669 } } }
I20260925 15:25:27.515672 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.515686 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.515882 7087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:27.516376 7087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:27.518124 7087 catalog_manager.cc:1347] Generated new cluster ID: 9662b1b804dc4bc596d85b50f43f4979
I20260925 15:25:27.518193 7087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:27.541923 7087 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:27.542667 7087 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:27.546070 7087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9c13a0705f646d888fcf2cd5f6a64e5: Generated new TSK 0
I20260925 15:25:27.546300 7087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:27.563289 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.558880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.1.163.126:44669)
0925 15:25:27.558953 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.558956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.559118 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:25:27.559171 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.559172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.559187 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.559277 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:25:27.559449 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.559956 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.561207 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.561437 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.561513 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.561730 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:25:27.563022 (+ 1292us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:25:27.563186 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 15:25:27.563371 7076 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:25:27.565485 7102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:27.558816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44669 (local address 127.0.0.1:59786)
0925 15:25:27.559012 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:27.559025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:27.559102 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:27.559291 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:27.559292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:27.559294 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:25:27.559359 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.559362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.560085 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.560086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.561102 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.561105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.561538 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.561539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.561602 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.561609 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:25:27.563148 (+ 1539us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:25:27.565385 (+ 2237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.163.126:44669: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2015,"thread_start_us":120,"threads_started":1}
I20260925 15:25:27.565991 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7006
E20260925 15:25:27.576676 7103 webserver.cc:533] Webserver: sslize failed:
E20260925 15:25:27.576782 7103 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T15:25:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:27Z Disabled control of system clock
I20260925 15:25:27.612957 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36747
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:39977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36747
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/testchainca.cert with env {}
W20260925 15:25:27.696720 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:27.696966 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:27.696993 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:27.698323 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:27.698385 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:27.698398 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:27.698411 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:27.698424 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:25:27.698437 7107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:25:27.698449 7107 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:25:27.700125 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:39977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:27.700414 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:27.700690 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:27.706683 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:27.706707 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:27.706732 7107 server_base.cc:1034] running on GCE node
W20260925 15:25:27.706714 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:27.707437 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:27.707732 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:27.708966 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790349927708950 us; error 33 us; skew 500 ppm
I20260925 15:25:27.710564 7107 webserver.cc:460] Webserver started at http://127.1.163.126:38363/ using document root <none> and password file <none>
I20260925 15:25:27.710866 7107 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:27.710932 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:27.711081 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:27.712375 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "b1d2de2d12ac4287a433d4549e4fb5b7"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "aef5c18b3d61222da64475705f9fd89e"
server_key_iv: "78e342d6f518fbb53eeb2de37859f002"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.712837 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "b1d2de2d12ac4287a433d4549e4fb5b7"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "aef5c18b3d61222da64475705f9fd89e"
server_key_iv: "78e342d6f518fbb53eeb2de37859f002"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.714351 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:27.715287 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:27.715750 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:27.715888 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "b1d2de2d12ac4287a433d4549e4fb5b7"
format_stamp: "Formatted at 2026-09-25 15:25:27 on dist-test-slave-hvv5"
server_key: "aef5c18b3d61222da64475705f9fd89e"
server_key_iv: "78e342d6f518fbb53eeb2de37859f002"
server_key_version: "encryptionkey@0"
I20260925 15:25:27.715975 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:27.729429 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:27.729786 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:27.729948 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:27.733987 7107 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36747
I20260925 15:25:27.734030 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36747 every 8 connection(s)
I20260925 15:25:27.734416 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:27.735584 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:27.737986 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7107
I20260925 15:25:27.738117 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:27.738075 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7: Bootstrap starting.
I20260925 15:25:27.738330 1677 external_mini_cluster.cc:1428] Setting key 84dfeba1174b08078c6e5f5a75b5f2b4
I20260925 15:25:27.738876 7174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:27.739218 7174 log.cc:824] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:27.740005 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7: No bootstrap required, opened a new log
I20260925 15:25:27.742456 7174 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } }
I20260925 15:25:27.742604 7174 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:27.742627 7174 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d2de2d12ac4287a433d4549e4fb5b7, State: Initialized, Role: FOLLOWER
I20260925 15:25:27.742771 7174 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } }
I20260925 15:25:27.742864 7174 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:27.742921 7174 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:27.742971 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:27.743757 7174 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } }
I20260925 15:25:27.743872 7174 leader_election.cc:302] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1d2de2d12ac4287a433d4549e4fb5b7; no voters:
I20260925 15:25:27.744064 7174 leader_election.cc:288] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:27.744259 7180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:27.744366 7174 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:27.744436 7180 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [term 1 LEADER]: Becoming Leader. State: Replica: b1d2de2d12ac4287a433d4549e4fb5b7, State: Running, Role: LEADER
I20260925 15:25:27.744527 7180 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } }
I20260925 15:25:27.744983 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1d2de2d12ac4287a433d4549e4fb5b7. Latest consensus state: current_term: 1 leader_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } } }
I20260925 15:25:27.744992 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d2de2d12ac4287a433d4549e4fb5b7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36747 } } }
I20260925 15:25:27.745085 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.745087 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:27.745904 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.738879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.1.163.126:36747)
0925 15:25:27.739146 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.739156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.739194 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:25:27.739800 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.739810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.739899 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.740060 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:27.740470 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.741898 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.742959 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.743570 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.743704 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.744726 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:27.744730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:27.744730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:27.744979 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:27.745031 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:27.745050 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:27.745265 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:27.745545 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:25:27.745648 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260925 15:25:27.745939 7188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:27.746909 7188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:27.747337 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:25:27.748661 7188 catalog_manager.cc:1347] Generated new cluster ID: ec1415d881b84aebb618ae51b1da16d3
I20260925 15:25:27.748718 7188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:27.754529 7188 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:27.755228 7188 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:27.768414 7188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1d2de2d12ac4287a433d4549e4fb5b7: Generated new TSK 0
I20260925 15:25:27.768673 7188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:27.809995 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:27.780655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.1.163.126:36747)
0925 15:25:27.780706 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:27.780708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:27.780740 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:25:27.780833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:27.780834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:27.780848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:27.780928 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:25:27.781164 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.782122 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.783064 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.783434 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.783520 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.783756 (+ 236us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:25:27.809659 (+ 25903us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:25:27.809913 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40394: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:25:27.810060 7203 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:27.780499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36747 (local address 127.0.0.1:40394)
0925 15:25:27.780678 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:27.780690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:27.780764 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:27.780927 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:27.780927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:27.780929 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:25:27.780987 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.780989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.782252 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.782253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.782937 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:27.782941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:27.783553 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:27.783554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:27.783621 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:27.783627 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:25:27.809830 (+ 26203us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:25:27.809952 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.163.126:36747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
W20260925 15:25:27.810089 7177 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40394: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 15:25:27.810338 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7107
2026-09-25T15:25:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (222 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:28 dist-test-slave-hvv5 krb5kdc[7215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:28 dist-test-slave-hvv5 krb5kdc[7215](info): set up 2 sockets
Sep 25 15:25:28 dist-test-slave-hvv5 krb5kdc[7215](info): commencing operation
krb5kdc: starting...
W20260925 15:25:29.104089 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 15:25:29.312454 1677 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.208s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 15:25:29.483523 1677 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.171s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 15:25:29.753296 1677 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.270s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/test-user.keytab.
W20260925 15:25:30.132649 1677 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.379s user 0.002s sys 0.000s
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:30.185217 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:39193
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:30.273310 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:30.273597 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:30.273630 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:30.274878 7228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:30.274921 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:30.274935 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:30.274948 7228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:30.274960 7228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:30.276490 7228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:30.276742 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:30.276932 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:30.282282 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.282261 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.282387 7228 server_base.cc:1034] running on GCE node
W20260925 15:25:30.282480 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.282757 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:30.283047 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:30.284252 7228 hybrid_clock.cc:648] HybridClock initialized: now 1790349930284239 us; error 36 us; skew 500 ppm
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.297078 7228 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:30.297657 7228 webserver.cc:460] Webserver started at http://127.1.163.126:33001/ using document root <none> and password file <none>
I20260925 15:25:30.297909 7228 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:30.297983 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:30.298142 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:30.299381 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "e7fea78a98194f98840035c1631086b6"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "a7ff349d7c55ffc50b0019f621d2a98f"
server_key_iv: "6a7413c4e00bdca7507e20538f424b27"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.299875 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "e7fea78a98194f98840035c1631086b6"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "a7ff349d7c55ffc50b0019f621d2a98f"
server_key_iv: "6a7413c4e00bdca7507e20538f424b27"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.301510 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:25:30.302443 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.302677 7228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:30.302845 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "e7fea78a98194f98840035c1631086b6"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "a7ff349d7c55ffc50b0019f621d2a98f"
server_key_iv: "6a7413c4e00bdca7507e20538f424b27"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.302942 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:30.318316 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:30.320616 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:30.320793 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:30.325349 7228 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:39193
I20260925 15:25:30.325392 7295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:39193 every 8 connection(s)
I20260925 15:25:30.325851 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:30.326864 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:30.329639 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6: Bootstrap starting.
I20260925 15:25:30.330305 7296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:30.330716 7296 log.cc:824] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:30.330792 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7228
I20260925 15:25:30.330885 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:30.331097 1677 external_mini_cluster.cc:1428] Setting key 8dd51eb7567fd5ef212a33dc0bf883a5
I20260925 15:25:30.331679 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6: No bootstrap required, opened a new log
I20260925 15:25:30.333860 7296 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } }
I20260925 15:25:30.334072 7296 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:30.334112 7296 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fea78a98194f98840035c1631086b6, State: Initialized, Role: FOLLOWER
I20260925 15:25:30.334266 7296 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } }
I20260925 15:25:30.334367 7296 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:30.334403 7296 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:30.334451 7296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:30.335264 7296 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } }
I20260925 15:25:30.335377 7296 leader_election.cc:302] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7fea78a98194f98840035c1631086b6; no voters:
I20260925 15:25:30.335541 7296 leader_election.cc:288] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:30.335628 7301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:30.335850 7301 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [term 1 LEADER]: Becoming Leader. State: Replica: e7fea78a98194f98840035c1631086b6, State: Running, Role: LEADER
I20260925 15:25:30.335847 7296 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:30.336066 7301 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } }
I20260925 15:25:30.336432 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7fea78a98194f98840035c1631086b6. Latest consensus state: current_term: 1 leader_uuid: "e7fea78a98194f98840035c1631086b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } } }
I20260925 15:25:30.336441 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7fea78a98194f98840035c1631086b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fea78a98194f98840035c1631086b6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39193 } } }
I20260925 15:25:30.336692 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:30.336894 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:30.337255 7310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:30.337929 7310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:30.339818 7310 catalog_manager.cc:1347] Generated new cluster ID: 7918db75e1894dcbb8c56f28cbd7cfdc
I20260925 15:25:30.339871 7310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:30.344753 7310 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:30.345084 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.331654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.1.163.126:39193)
0925 15:25:30.331876 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.331886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.331925 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:25:30.333071 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.333082 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.333154 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.333319 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.334023 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.335725 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.336963 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.337658 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.337832 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.340841 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.340845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.340845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.340859 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.343011 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.343447 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.343450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.343451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.343567 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.343795 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.343797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.343797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.344132 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.344477 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.344668 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.344860 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260925 15:25:30.345348 7310 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:30.348174 7310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7fea78a98194f98840035c1631086b6: Generated new TSK 0
I20260925 15:25:30.348357 7310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:30.383682 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:30.478926 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:30.479166 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:30.479207 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:30.480676 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:30.480759 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:30.480806 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:30.482424 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:30.482705 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:30.482947 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:30.488273 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.488273 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.488299 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.489015 7322 server_base.cc:1034] running on GCE node
I20260925 15:25:30.489252 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:30.489528 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:30.490720 7322 hybrid_clock.cc:648] HybridClock initialized: now 1790349930490703 us; error 33 us; skew 500 ppm
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.493521 7322 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:30.493992 7322 webserver.cc:460] Webserver started at http://127.1.163.65:39291/ using document root <none> and password file <none>
I20260925 15:25:30.494191 7322 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:30.494233 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:30.494326 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:30.495361 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "afd525183b774cd48c67acdc13cd90f3"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "cba9d8d07a34d2b31df29769ecfbe926"
server_key_iv: "19428b200086e4627d92eddcd1ca3173"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.495668 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "afd525183b774cd48c67acdc13cd90f3"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "cba9d8d07a34d2b31df29769ecfbe926"
server_key_iv: "19428b200086e4627d92eddcd1ca3173"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.497226 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 15:25:30.498039 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.498245 7322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:30.498320 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "afd525183b774cd48c67acdc13cd90f3"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "cba9d8d07a34d2b31df29769ecfbe926"
server_key_iv: "19428b200086e4627d92eddcd1ca3173"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.498417 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:30.511740 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:30.514230 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:30.514387 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:30.514660 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:30.514994 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:30.515029 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.515054 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:30.515067 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.521217 7322 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:46327
I20260925 15:25:30.521304 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:46327 every 8 connection(s)
I20260925 15:25:30.521646 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:30.529678 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7322
I20260925 15:25:30.529800 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:30.529996 1677 external_mini_cluster.cc:1428] Setting key e183f2fa501ef89937d8bd43c6d1c30c
I20260925 15:25:30.531790 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.522510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:55431 (local address 127.1.163.126:39193)
0925 15:25:30.522597 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.522599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.522808 (+ 209us) server_negotiation.cc:407] Connection header received
0925 15:25:30.523857 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.523859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.523873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.523923 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.524661 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.525197 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.526225 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.526467 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.526553 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.528798 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.528800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.528801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.528810 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.530278 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.530710 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.530711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.530712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.530773 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.531052 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.531053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.531053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.531203 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.531276 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.531544 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.531645 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:25:30.531816 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.522452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39193 (local address 127.1.163.65:55431)
0925 15:25:30.522665 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:30.522709 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:30.523734 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:30.523997 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:30.524001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:30.524281 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:30.524489 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.524494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.525309 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.525310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.526127 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.526129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.526565 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.526567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.526699 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.527261 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:30.527270 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:30.528696 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:30.530385 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.530392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.530395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.530620 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.530859 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.530860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.530861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.530981 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.531271 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:30.531275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:30.531415 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 15:25:30.531502 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:25:30.531642 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260925 15:25:30.532644 7451 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39193
I20260925 15:25:30.532761 7451 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:30.532971 7451 heartbeater.cc:511] Master 127.1.163.126:39193 requested a full tablet report, sending...
I20260925 15:25:30.533519 7260 ts_manager.cc:194] Registered new tserver with Master: afd525183b774cd48c67acdc13cd90f3 (127.1.163.65:46327)
I20260925 15:25:30.534137 7260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:55431
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:30.557662 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:30.641678 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:30.641966 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:30.642014 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:30.643285 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:30.643366 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:30.643411 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:30.645068 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:30.645370 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:30.645622 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:30.651188 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.651196 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.651196 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.651482 7456 server_base.cc:1034] running on GCE node
I20260925 15:25:30.651749 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:30.652046 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:30.653213 7456 hybrid_clock.cc:648] HybridClock initialized: now 1790349930653190 us; error 55 us; skew 500 ppm
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.655997 7456 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:30.656519 7456 webserver.cc:460] Webserver started at http://127.1.163.66:46837/ using document root <none> and password file <none>
I20260925 15:25:30.656836 7456 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:30.656910 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:30.657063 7456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:30.658556 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "1e2e46fa15e94fc6b3023eae9ff06aac"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "0d67e80d8af4a1100c5752ac707b185c"
server_key_iv: "817ee3569d08abc99d9f4a20432d51bd"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.659011 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "1e2e46fa15e94fc6b3023eae9ff06aac"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "0d67e80d8af4a1100c5752ac707b185c"
server_key_iv: "817ee3569d08abc99d9f4a20432d51bd"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.660528 7456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:25:30.661404 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.661576 7456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:30.661680 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "1e2e46fa15e94fc6b3023eae9ff06aac"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "0d67e80d8af4a1100c5752ac707b185c"
server_key_iv: "817ee3569d08abc99d9f4a20432d51bd"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.661772 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:30.675312 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:30.677840 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:30.678026 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:30.678284 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:30.678665 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:30.678730 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.678776 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:30.678802 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.685524 7456 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34203
I20260925 15:25:30.685559 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34203 every 8 connection(s)
I20260925 15:25:30.685918 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:30.693238 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7456
I20260925 15:25:30.693353 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:30.693593 1677 external_mini_cluster.cc:1428] Setting key 274dc227a0de8b3a267d78865a513276
I20260925 15:25:30.696879 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.686704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:33983 (local address 127.1.163.126:39193)
0925 15:25:30.686773 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.686775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.686996 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:25:30.687953 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.687954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.687964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.688030 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.688713 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.689223 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.690271 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.690483 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.690541 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.693085 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.693087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.693087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.693098 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.694988 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.695489 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.695491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.695492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.695560 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.695828 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.695829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.695830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.696009 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.696080 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.696624 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.696735 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:25:30.697053 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.686686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39193 (local address 127.1.163.66:33983)
0925 15:25:30.686856 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:30.686901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:30.687848 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:30.688086 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:30.688090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:30.688345 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:30.688602 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.688608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.689328 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.689330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.690189 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.690192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.690565 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.690566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.690676 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.691321 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:30.691330 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:30.692865 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:30.695096 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.695104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.695106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.695404 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.695636 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.695637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.695637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.695761 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.696147 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:30.696155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:30.696375 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 15:25:30.696530 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:25:30.696775 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260925 15:25:30.697755 7585 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39193
I20260925 15:25:30.697860 7585 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:30.698071 7585 heartbeater.cc:511] Master 127.1.163.126:39193 requested a full tablet report, sending...
I20260925 15:25:30.698510 7260 ts_manager.cc:194] Registered new tserver with Master: 1e2e46fa15e94fc6b3023eae9ff06aac (127.1.163.66:34203)
I20260925 15:25:30.698911 7260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:33983
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:30.724157 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:30.811762 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:30.811980 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:30.812008 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:30.813196 7590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:30.813238 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:30.813265 7590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:30.815168 7590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:30.815424 7590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:30.815623 7590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:30.820850 7598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:30.820852 7595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.821177 7590 server_base.cc:1034] running on GCE node
W20260925 15:25:30.820849 7596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:30.821478 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:30.821736 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:30.822892 7590 hybrid_clock.cc:648] HybridClock initialized: now 1790349930822899 us; error 65 us; skew 500 ppm
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.825673 7590 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:30.826097 7590 webserver.cc:460] Webserver started at http://127.1.163.67:45191/ using document root <none> and password file <none>
I20260925 15:25:30.826367 7590 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:30.826439 7590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:30.826607 7590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:30.827706 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "1ad09b94e4f1478aa68a7b1ad652b9f2"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "c3ba2e40a7db252d6a3befee66706ff6"
server_key_iv: "2d45fe325f9f76c2ef1d9798cb70ed74"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.828081 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "1ad09b94e4f1478aa68a7b1ad652b9f2"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "c3ba2e40a7db252d6a3befee66706ff6"
server_key_iv: "2d45fe325f9f76c2ef1d9798cb70ed74"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.829771 7590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 15:25:30.830600 7605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.830796 7590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:30.830904 7590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "1ad09b94e4f1478aa68a7b1ad652b9f2"
format_stamp: "Formatted at 2026-09-25 15:25:30 on dist-test-slave-hvv5"
server_key: "c3ba2e40a7db252d6a3befee66706ff6"
server_key_iv: "2d45fe325f9f76c2ef1d9798cb70ed74"
server_key_version: "encryptionkey@0"
I20260925 15:25:30.830981 7590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:30.849102 7590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:30.851577 7590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:30.851750 7590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:30.852026 7590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:30.852380 7590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:30.852437 7590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.852480 7590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:30.852512 7590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:30.858865 7590 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:43819
I20260925 15:25:30.858946 7718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:43819 every 8 connection(s)
I20260925 15:25:30.859305 7590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:30.859817 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7590
I20260925 15:25:30.859913 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:30.860103 1677 external_mini_cluster.cc:1428] Setting key e990046a8df10f074011c5c44c5a45dc
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:30.869983 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.860153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:52551 (local address 127.1.163.126:39193)
0925 15:25:30.860221 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.860223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.860465 (+ 242us) server_negotiation.cc:407] Connection header received
0925 15:25:30.861648 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.861649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.861663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.861726 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.862391 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.862896 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.863919 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.864115 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.864202 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.866990 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.866991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.866992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.866999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.868412 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.868931 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.868933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.868933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.869027 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.869297 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.869298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.869298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.869425 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.869505 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.869792 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.869875 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:25:30.870182 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.860153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39193 (local address 127.1.163.67:52551)
0925 15:25:30.860341 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:30.860388 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:30.861531 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:30.861801 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:30.861805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:30.862080 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:30.862283 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.862288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.862997 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.862998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.863824 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:30.863827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.864214 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.864215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:30.864351 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.865110 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:30.865119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:30.866702 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:30.868561 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.868570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.868572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.868831 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.869104 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:30.869105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:30.869105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:30.869221 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:30.869494 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:30.869501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:30.869652 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:25:30.869759 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:25:30.869926 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260925 15:25:30.870669 7719 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39193
I20260925 15:25:30.870764 7719 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:30.871030 7719 heartbeater.cc:511] Master 127.1.163.126:39193 requested a full tablet report, sending...
I20260925 15:25:30.871433 7259 ts_manager.cc:194] Registered new tserver with Master: 1ad09b94e4f1478aa68a7b1ad652b9f2 (127.1.163.67:43819)
I20260925 15:25:30.871771 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:25:30.871956 7259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:52551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/test-admin.keytab.
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.890884 1677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:25:30.892663 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:30.891863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.1.163.65:46327)
0925 15:25:30.892033 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.892046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.892057 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:30.892085 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.892087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.892099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.892173 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.892534 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55000: BlockingRecv error: recv got EOF from 127.0.0.1:55000 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
W20260925 15:25:30.892791 1677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.893836 1677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.894942 1677 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:30.903690 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.895580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55012 (local address 127.1.163.65:46327)
0925 15:25:30.895658 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.895660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.895686 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:25:30.895744 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.895745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.895757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.895815 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.896420 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.897245 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.897984 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.898317 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.898383 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.900725 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.900735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.900735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.900751 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.902553 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.902928 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.902931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.902931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.903010 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.903205 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.903206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.903206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.903346 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.903454 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.903540 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.903610 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:25:30.905735 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:30.905086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.1.163.65:46327)
0925 15:25:30.905152 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.905154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.905225 (+ 71us) server_negotiation.cc:407] Connection header received
0925 15:25:30.905298 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.905299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.905311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.905382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.905680 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55024: BlockingRecv error: recv got EOF from 127.0.0.1:55024 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:25:30.905874 1677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.906981 1677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.907912 1677 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:30.915439 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.908500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.1.163.65:46327)
0925 15:25:30.908580 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.908582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.908637 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:25:30.908714 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.908715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.908728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.908797 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.909206 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.909705 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.910451 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.910661 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.910750 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.913040 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.913042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.913042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.913049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.914344 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.914661 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.914662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.914662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.914711 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.914906 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.914906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.914907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.914999 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.915078 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.915230 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.915294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:25:30.917239 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:30.916524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.1.163.65:46327)
0925 15:25:30.916641 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.916643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.916665 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:25:30.916722 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.916723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.916731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.916772 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.917202 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55042: BlockingRecv error: recv got EOF from 127.0.0.1:55042 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 15:25:30.917333 1677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349904034090-1677-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.918366 1677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:30.919216 1677 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:25:30 dist-test-slave-hvv5 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.65@KRBTEST.COM
I20260925 15:25:30.926852 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:30.919749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.1.163.65:46327)
0925 15:25:30.919804 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:30.919805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:30.919850 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:25:30.919912 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:30.919913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:30.919920 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:30.919975 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:30.920343 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.920880 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.921554 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:30.921740 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:30.921789 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:30.924077 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:30.924079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:30.924079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:30.924086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:30.925666 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.926039 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.926040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.926040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.926095 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:30.926298 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:30.926299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:30.926299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:30.926427 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:30.926536 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:30.926636 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:25:30.926701 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:25:30.927610 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7322
I20260925 15:25:30.932531 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7456
I20260925 15:25:30.937132 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7590
I20260925 15:25:30.941628 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7228
2026-09-25T15:25:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2951 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:31 dist-test-slave-hvv5 krb5kdc[7739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:31 dist-test-slave-hvv5 krb5kdc[7739](info): set up 2 sockets
Sep 25 15:25:31 dist-test-slave-hvv5 krb5kdc[7739](info): commencing operation
krb5kdc: starting...
W20260925 15:25:31.988881 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab.
Entry for principal oryx/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab.
I20260925 15:25:32.116241 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36199
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab
--principal=oryx/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:32.202270 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:32.202553 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:32.202591 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:32.204955 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:32.205024 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:32.205050 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:32.205075 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:32.205137 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:32.208154 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab
--principal=oryx/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:32.208463 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:32.208922 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:32.215164 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.215200 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.215400 7752 server_base.cc:1034] running on GCE node
W20260925 15:25:32.215205 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.215983 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:32.216336 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:32.217530 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790349932217496 us; error 45 us; skew 500 ppm
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:32.220336 7752 init.cc:377] Logged in from keytab as oryx/127.1.163.126@KRBTEST.COM (short username oryx)
I20260925 15:25:32.220984 7752 webserver.cc:460] Webserver started at http://127.1.163.126:45307/ using document root <none> and password file <none>
I20260925 15:25:32.221379 7752 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:32.221477 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:32.221644 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:32.223019 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "c5bce094271e4f6e9cc745f750f9295f"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "b67235a85a18d211a9730568cb46f694"
server_key_iv: "52d01c92972d57e0879f1ba490ed0e76"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.223541 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "c5bce094271e4f6e9cc745f750f9295f"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "b67235a85a18d211a9730568cb46f694"
server_key_iv: "52d01c92972d57e0879f1ba490ed0e76"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.225451 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:25:32.226434 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.226653 7752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:32.226765 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "c5bce094271e4f6e9cc745f750f9295f"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "b67235a85a18d211a9730568cb46f694"
server_key_iv: "52d01c92972d57e0879f1ba490ed0e76"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.226854 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:32.252313 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:32.254400 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:32.254514 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:32.258586 7752 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36199
I20260925 15:25:32.258643 7819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36199 every 8 connection(s)
I20260925 15:25:32.259025 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:32.260069 7820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:32.261646 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7752
I20260925 15:25:32.261755 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:32.261961 1677 external_mini_cluster.cc:1428] Setting key 9c581f827032f83b83592f42e16cdcbe
I20260925 15:25:32.262971 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Bootstrap starting.
I20260925 15:25:32.263581 7820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:32.263876 7820 log.cc:824] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:32.264722 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: No bootstrap required, opened a new log
I20260925 15:25:32.267267 7820 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } }
I20260925 15:25:32.267483 7820 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:32.267535 7820 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bce094271e4f6e9cc745f750f9295f, State: Initialized, Role: FOLLOWER
I20260925 15:25:32.267714 7820 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } }
I20260925 15:25:32.268167 7820 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:32.268242 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:32.268606 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:32.269306 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } }
I20260925 15:25:32.269433 7820 leader_election.cc:302] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5bce094271e4f6e9cc745f750f9295f; no voters:
I20260925 15:25:32.269626 7820 leader_election.cc:288] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:32.269685 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:32.269840 7825 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 LEADER]: Becoming Leader. State: Replica: c5bce094271e4f6e9cc745f750f9295f, State: Running, Role: LEADER
I20260925 15:25:32.269923 7820 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:32.269965 7825 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } }
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:32.270315 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bce094271e4f6e9cc745f750f9295f. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } }
I20260925 15:25:32.270404 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:32.270700 7832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:32.271435 7832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:32.271802 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } }
I20260925 15:25:32.271916 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:32.273026 7842 mvcc.cc:205] Tried to move back new op lower bound from 7333273322588045312 to 7333273322578124800. Current Snapshot: MvccSnapshot[applied={T|T < 7333273322588045312}]
I20260925 15:25:32.273856 7832 catalog_manager.cc:1347] Generated new cluster ID: 6abbc51a20e748c7b893c5f4a1347ba6
I20260925 15:25:32.273908 7832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:32.274665 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.262525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.1.163.126:36199)
0925 15:25:32.262724 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.262734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.262768 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:25:32.263248 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.263256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.263306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.263461 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.264196 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.265692 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.266539 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.267455 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.267640 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.270763 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.270768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.270768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.270782 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.272916 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.273370 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.273373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.273373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.273456 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.273686 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.273687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.273687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.274008 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.274145 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.274339 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.274470 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260925 15:25:32.279801 7832 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:32.280362 7832 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:32.282924 7832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Generated new TSK 0
I20260925 15:25:32.283110 7832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
I20260925 15:25:32.320369 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:32.412420 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:32.412719 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:32.412761 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:32.413976 7846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:32.414047 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:32.414093 7846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:32.415602 7846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:32.415882 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:32.416127 7846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:32.422318 7852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.422444 7846 server_base.cc:1034] running on GCE node
W20260925 15:25:32.422539 7851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.422328 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.422878 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:32.423179 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:32.424345 7846 hybrid_clock.cc:648] HybridClock initialized: now 1790349932424309 us; error 53 us; skew 500 ppm
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:32.426908 7846 init.cc:377] Logged in from keytab as oryx/127.1.163.65@KRBTEST.COM (short username oryx)
I20260925 15:25:32.427371 7846 webserver.cc:460] Webserver started at http://127.1.163.65:41281/ using document root <none> and password file <none>
I20260925 15:25:32.427624 7846 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:32.427695 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:32.427847 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:32.429013 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "b336feea90e546059515bc83bffffcec"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "6bc72ac95bda6328497e7ea8b2b5590a"
server_key_iv: "e5fb7f63c9b1bdeb21704555e15fee99"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.429384 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "b336feea90e546059515bc83bffffcec"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "6bc72ac95bda6328497e7ea8b2b5590a"
server_key_iv: "e5fb7f63c9b1bdeb21704555e15fee99"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.430874 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:25:32.431740 7861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.431917 7846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:32.432025 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "b336feea90e546059515bc83bffffcec"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "6bc72ac95bda6328497e7ea8b2b5590a"
server_key_iv: "e5fb7f63c9b1bdeb21704555e15fee99"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.432102 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:32.450984 7846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:32.453323 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:32.453509 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:32.453804 7846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:32.454159 7846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:32.454224 7846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.454265 7846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:32.454319 7846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.461016 7846 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35985
I20260925 15:25:32.461069 7974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35985 every 8 connection(s)
I20260925 15:25:32.461418 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:32.466670 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7846
I20260925 15:25:32.466795 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:32.467063 1677 external_mini_cluster.cc:1428] Setting key 41ed00e371f0490263545482989f7320
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:32.472301 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.462275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:55329 (local address 127.1.163.126:36199)
0925 15:25:32.462373 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.462376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.462555 (+ 179us) server_negotiation.cc:407] Connection header received
0925 15:25:32.463100 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.463101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.463113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.463175 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.464183 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.464799 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.465836 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.466064 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.466154 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.468956 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.468958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.468958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.468968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.470744 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.471265 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.471267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.471267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.471328 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.471591 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.471592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.471593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.471738 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.471842 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.472113 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.472204 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:25:32.472568 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.462213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:55329)
0925 15:25:32.462415 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.462463 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.462987 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.463256 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.463260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.463858 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.464072 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.464077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.464906 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.464908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.465726 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.465728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.466167 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.466168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.466293 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.467058 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.467067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.468845 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.470850 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.470857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.470860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.471188 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.471403 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.471404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.471404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.471522 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.471824 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:32.471831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:32.471980 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 15:25:32.472094 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:25:32.472263 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260925 15:25:32.473132 7975 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:32.473256 7975 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:32.473443 7975 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:32.473999 7784 ts_manager.cc:194] Registered new tserver with Master: b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
I20260925 15:25:32.474627 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:55329
WARNING: no policy specified for oryx/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
I20260925 15:25:32.497459 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:32.579885 7980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:32.580152 7980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:32.580190 7980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:32.581429 7980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:32.581504 7980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:32.581553 7980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:32.583043 7980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:32.583333 7980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:32.583575 7980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:32.588876 7988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.588877 7986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.588925 7985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.589035 7980 server_base.cc:1034] running on GCE node
I20260925 15:25:32.589349 7980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:32.589608 7980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:32.590844 7980 hybrid_clock.cc:648] HybridClock initialized: now 1790349932590803 us; error 26 us; skew 500 ppm
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:32.593485 7980 init.cc:377] Logged in from keytab as oryx/127.1.163.66@KRBTEST.COM (short username oryx)
I20260925 15:25:32.593976 7980 webserver.cc:460] Webserver started at http://127.1.163.66:46005/ using document root <none> and password file <none>
I20260925 15:25:32.594177 7980 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:32.594218 7980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:32.594321 7980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:32.595378 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "85bb367b5ae8c6d02810f2c92a2bb958"
server_key_iv: "7a7faed0dde3d8861a60733518987048"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.595686 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "85bb367b5ae8c6d02810f2c92a2bb958"
server_key_iv: "7a7faed0dde3d8861a60733518987048"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.597087 7980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:25:32.597857 7995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.598064 7980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:32.598179 7980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "85bb367b5ae8c6d02810f2c92a2bb958"
server_key_iv: "7a7faed0dde3d8861a60733518987048"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.598269 7980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:32.625005 7980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:32.627050 7980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:32.627247 7980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:32.627508 7980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:32.627826 7980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:32.627884 7980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.627931 7980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:32.627956 7980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.634296 7980 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39311
I20260925 15:25:32.634335 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39311 every 8 connection(s)
I20260925 15:25:32.634680 7980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:32.642742 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 7980
I20260925 15:25:32.642866 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:32.643064 1677 external_mini_cluster.cc:1428] Setting key af911c5170c2ecfa023ad8e300019372
I20260925 15:25:32.644676 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.635450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:36955 (local address 127.1.163.126:36199)
0925 15:25:32.635521 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.635523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.635754 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:25:32.636277 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.636278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.636289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.636365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.637411 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.637920 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.638903 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.639090 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.639149 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.641616 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.641617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.641618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.641627 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.642997 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.643491 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.643492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.643493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.643560 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.643945 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.643946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.643946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.644086 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.644157 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.644461 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.644590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:25:32.644878 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.635456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:36955)
0925 15:25:32.635613 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.635668 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.636177 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.636478 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.636482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.637114 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.637320 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.637324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.638006 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.638007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.638817 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.638818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.639168 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.639169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.639274 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.639862 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.639870 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.641482 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.643095 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.643103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.643105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.643401 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.643654 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.643654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.643654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.643862 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.644159 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:32.644166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:32.644318 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 15:25:32.644425 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:25:32.644612 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260925 15:25:32.645381 8109 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:32.645479 8109 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:32.645678 8109 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:32.646040 7784 ts_manager.cc:194] Registered new tserver with Master: bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311)
I20260925 15:25:32.646457 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:36955
WARNING: no policy specified for oryx/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
I20260925 15:25:32.669389 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:32.753387 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:32.753600 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:32.753625 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:32.754863 8114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:32.754915 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:32.754944 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:32.756382 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:32.756644 8114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:32.756845 8114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:32.762784 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.762880 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.762952 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.763096 8114 server_base.cc:1034] running on GCE node
I20260925 15:25:32.763336 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:32.763614 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:32.764765 8114 hybrid_clock.cc:648] HybridClock initialized: now 1790349932764751 us; error 28 us; skew 500 ppm
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:32.767477 8114 init.cc:377] Logged in from keytab as oryx/127.1.163.67@KRBTEST.COM (short username oryx)
I20260925 15:25:32.768019 8114 webserver.cc:460] Webserver started at http://127.1.163.67:37387/ using document root <none> and password file <none>
I20260925 15:25:32.768290 8114 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:32.768371 8114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:32.768579 8114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:32.769958 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "dcd004cc9b77e505f23be3f0ba56cbac"
server_key_iv: "59db393926fea894058e11658155b7a5"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.770357 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "dcd004cc9b77e505f23be3f0ba56cbac"
server_key_iv: "59db393926fea894058e11658155b7a5"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.772396 8114 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:25:32.773339 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.773545 8114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:32.773654 8114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "dcd004cc9b77e505f23be3f0ba56cbac"
server_key_iv: "59db393926fea894058e11658155b7a5"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.773752 8114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:32.787120 8114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:32.789494 8114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:32.789680 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:32.789947 8114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:32.790288 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:32.790351 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.790401 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:32.790428 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.798175 8114 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:46707
I20260925 15:25:32.798269 8242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:46707 every 8 connection(s)
I20260925 15:25:32.798646 8114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:32.804796 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8114
I20260925 15:25:32.804917 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:32.805140 1677 external_mini_cluster.cc:1428] Setting key f6fa2ee6b15dcf2fd811c9da907ce186
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:32.808717 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.799544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:39437 (local address 127.1.163.126:36199)
0925 15:25:32.799595 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.799597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.799863 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:25:32.800284 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.800285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.800299 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.800373 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.801527 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.802032 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.803154 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.803343 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.803404 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.806129 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.806131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.806131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.806139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.807315 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.807721 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.807723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.807723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.807790 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.808044 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.808044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.808045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.808166 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.808251 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.808520 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.808623 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:32.808936 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.799459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:39437)
0925 15:25:32.799644 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.799687 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.800189 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.800435 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.800439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.801223 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.801432 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.801437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.802138 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.802140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.803067 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.803070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.803424 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.803425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.803530 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.804163 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.804186 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.806044 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.807414 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.807421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.807423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.807642 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.807859 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.807860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.807860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.807975 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.808233 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:32.808239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:32.808386 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 15:25:32.808500 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:25:32.808686 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 15:25:32.809468 8243 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:32.809590 8243 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:32.809816 8243 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:32.810241 7784 ts_manager.cc:194] Registered new tserver with Master: b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:32.810599 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:39437
I20260925 15:25:32.817200 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.163.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.125.keytab.
Entry for principal oryx/127.1.163.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.125.keytab.
I20260925 15:25:32.844736 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:45103
--webserver_interface=127.1.163.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.125.keytab
--principal=oryx/127.1.163.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36199,127.1.163.125:45103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:32.927523 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:32.927783 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:32.927809 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:32.929239 8248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:32.929291 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:32.929306 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:32.929319 8248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:32.929340 8248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:32.930953 8248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36199,127.1.163.125:45103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.125.keytab
--principal=oryx/127.1.163.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.163.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
W20260925 15:25:32.931114 8248 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.163.126:36199,127.1.163.125:45103'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:25:32.931249 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:32.931437 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:32.936735 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:32.936743 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.936776 8248 server_base.cc:1034] running on GCE node
W20260925 15:25:32.936743 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:32.937191 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:32.937522 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:32.938668 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790349932938656 us; error 28 us; skew 500 ppm
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:32.941442 8248 init.cc:377] Logged in from keytab as oryx/127.1.163.125@KRBTEST.COM (short username oryx)
I20260925 15:25:32.941962 8248 webserver.cc:460] Webserver started at http://127.1.163.125:44199/ using document root <none> and password file <none>
I20260925 15:25:32.942199 8248 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:32.942246 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:32.942389 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:32.943543 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/instance:
uuid: "3513a97b69314325830b8b2c75b0e6dd"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "43963ee8690416660427972a779b1009"
server_key_iv: "9b5cbef0773cdc7fea9dcf358d056b3c"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.943910 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal/instance:
uuid: "3513a97b69314325830b8b2c75b0e6dd"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "43963ee8690416660427972a779b1009"
server_key_iv: "9b5cbef0773cdc7fea9dcf358d056b3c"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.945359 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:32.946121 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:32.946352 8248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:32.946422 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "3513a97b69314325830b8b2c75b0e6dd"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "43963ee8690416660427972a779b1009"
server_key_iv: "9b5cbef0773cdc7fea9dcf358d056b3c"
server_key_version: "encryptionkey@0"
I20260925 15:25:32.946525 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:32.954993 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:32.957218 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:32.957387 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:32.961721 8248 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.125:45103
I20260925 15:25:32.961778 8315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.125:45103 every 8 connection(s)
I20260925 15:25:32.962221 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
I20260925 15:25:32.963225 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:32.964959 8316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:25:32.969801 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8248
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.125@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:32.974821 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.965747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.1.163.126:36199)
0925 15:25:32.965791 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.965793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.965993 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:25:32.966463 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.966464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.966474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.966528 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.967121 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.967623 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.968756 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.968964 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.969021 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.971522 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.971524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.971524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.971533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.973173 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.973550 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.973551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.973552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.973620 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.974008 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.974009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.974009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.974184 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.974259 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.974615 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.974717 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:32.974992 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.965713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58688)
0925 15:25:32.965871 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.965904 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.966356 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.966578 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.966581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.966837 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.967023 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.967026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.967722 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.967723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.968677 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.968679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.969053 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.969054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.969163 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.969730 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.969746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.971451 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.973256 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.973261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.973263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.973477 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.973775 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.973776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.973776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.973924 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.974273 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:32.974278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:32.974441 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:25:32.974568 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:25:32.974757 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
I20260925 15:25:32.976334 8316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:25:32.979910 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.970590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.1.163.125:45103)
0925 15:25:32.970714 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.970718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.970732 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:25:32.970749 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.970751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.970767 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.970848 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.971637 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.972632 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.973679 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.974119 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.974187 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.976885 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.976893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.976894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.976901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.978673 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.979039 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.979042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.979042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.979105 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.979310 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.979311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.979311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.979458 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.979637 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.979684 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.979799 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
Sep 25 15:25:32 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.125@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
I20260925 15:25:32.984273 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.976656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:51926)
0925 15:25:32.976878 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.976891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.976946 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.977302 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.977303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.977469 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.977512 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.977513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.978205 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.978206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.978738 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.978740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.979124 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.979125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.979187 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.979596 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.979601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.981567 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.983213 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.983214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.983215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.983381 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.983598 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.983598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.983599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.983651 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.983905 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:32.983906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:32.983944 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:25:32.984012 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:25:32.984154 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":17}
I20260925 15:25:32.984351 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.976731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.1.163.125:45103)
0925 15:25:32.976897 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.976899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.976930 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:25:32.977004 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.977004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.977015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.977292 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.977598 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.978122 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.978827 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.979043 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.979125 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.981659 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.981660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.981661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.981667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.983112 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.983474 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.983475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.983476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.983529 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.983721 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.983723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.983723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.983835 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:32.983917 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:32.984040 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:25:32.984200 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260925 15:25:32.985633 8316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Bootstrap starting.
I20260925 15:25:32.986301 8316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:32.986570 8316 log.cc:824] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:32.987252 8316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: No bootstrap required, opened a new log
I20260925 15:25:32.988715 8316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } }
I20260925 15:25:32.988837 8316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:32.988860 8316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3513a97b69314325830b8b2c75b0e6dd, State: Initialized, Role: FOLLOWER
I20260925 15:25:32.988950 8316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } }
I20260925 15:25:32.989180 8323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } } }
I20260925 15:25:32.989255 8316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:32.989269 8323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:25:32.990892 8334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:25:32.990962 8334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:25:32.992585 8248 master_runner.cc:428] Detected that this master 3513a97b69314325830b8b2c75b0e6dd is joining an existing cluster
I20260925 15:25:32.992636 8248 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.163.125:45103
I20260925 15:25:33.000494 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.993814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.1.163.125:45103)
0925 15:25:32.993874 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.993876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.994040 (+ 164us) server_negotiation.cc:407] Connection header received
0925 15:25:32.994103 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.994104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.994113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.994168 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.994535 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.995034 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.995787 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.996009 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.996096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.997492 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.997494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.997494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.997501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.999163 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999538 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.999539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.999539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.999611 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999806 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.999807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.999808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:32.999973 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.000047 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.000254 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.000418 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":42}
I20260925 15:25:33.000756 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.993635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:51940)
0925 15:25:32.993984 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.993991 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.994048 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.994183 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.994184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.994374 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.994424 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.994425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.995119 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.995120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.995702 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.995704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.996111 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.996111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.996168 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.996918 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.996922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.997404 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.999262 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.999264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.999451 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:32.999682 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.999682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.999732 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.000048 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.000078 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.000126 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.000231 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.000350 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":79,"threads_started":1}
I20260925 15:25:33.000907 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.993737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.1.163.126:36199)
0925 15:25:32.993781 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:32.993783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:32.993882 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:25:32.993949 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:32.993950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:32.993961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:32.994021 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:32.994545 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.995329 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.996452 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.996778 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.996834 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.997933 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:32.997935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:32.997935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:32.997943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:32.999554 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999957 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:32.999958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:32.999958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.000040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.000262 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.000264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.000265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.000410 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.000455 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.000731 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.000813 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:33.001260 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:32.993628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58696)
0925 15:25:32.993795 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:32.993806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:32.993896 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:32.994072 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:32.994073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:32.994326 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:32.994446 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.994448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.995430 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.995431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.996335 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:32.996338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:32.996883 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:32.996884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:32.996965 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:32.997417 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:32.997422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:32.997819 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:32.999644 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:32.999645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:32.999645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:32.999882 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.000111 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.000112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.000112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.000180 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.000506 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.000507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.000608 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.001062 (+ 454us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.001159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.126@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
I20260925 15:25:33.011138 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.003629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.1.163.125:45103)
0925 15:25:33.003671 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.003673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.003793 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:25:33.003891 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.003892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.003900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.003942 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.004431 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.004956 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.006060 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.006232 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.006296 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.008484 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.008485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.008486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.008492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.009965 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.010410 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.010411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.010412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.010469 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.010663 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.010664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.010664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.010761 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.010810 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.011006 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.011071 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:33.011226 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.003546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:51948)
0925 15:25:33.003699 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.003740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.003817 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.003955 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.003958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.004299 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.004360 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.004362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.005031 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.005032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.005986 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.005989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.006309 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.006310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.006392 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.006815 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.006820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.008395 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.010047 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.010054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.010057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.010337 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.010533 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.010534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.010534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.010603 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.010824 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.010832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.010889 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.010981 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.011106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260925 15:25:33.011816 7784 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.163.125:45103: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd"
member_type: NON_VOTER
last_known_addr {
host: "127.1.163.125"
port: 45103
}
attrs {
promote: true
}
}
dest_uuid: "c5bce094271e4f6e9cc745f750f9295f"
cas_config_opid_index: -1
I20260925 15:25:33.012176 7784 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } }
I20260925 15:25:33.012722 8343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } }
I20260925 15:25:33.013051 8290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:33.013852 8343 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.163.125:45103
I20260925 15:25:33.013955 8343 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } }
I20260925 15:25:33.014027 8343 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.014039 8290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:25:33.014082 8248 master.cc:537] Master@127.1.163.125:45103 shutting down...
I20260925 15:25:33.014464 8344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:25:33.015345 7768 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 15:25:33.015784 7768 consensus_peers.cc:487] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f -> Peer 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125:45103): Couldn't send request to peer 3513a97b69314325830b8b2c75b0e6dd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:25:33.016721 8248 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:33.016819 8248 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:33.016876 8248 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: stopping tablet replica
I20260925 15:25:33.019846 8248 master.cc:552] Master@127.1.163.125:45103 shutdown complete.
I20260925 15:25:33.019897 8248 master_runner.cc:305] Clearing existing system tablet
I20260925 15:25:33.020028 8248 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.020076 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.020784 8248 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260925 15:25:33.021405 8348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.021567 8248 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.021634 8248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:33.021653 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "3513a97b69314325830b8b2c75b0e6dd"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "43963ee8690416660427972a779b1009"
server_key_iv: "9b5cbef0773cdc7fea9dcf358d056b3c"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.022226 8248 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:33.023545 8248 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:25:33.023602 8248 log.cc:1196] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:25:33.023818 8248 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Deleting consensus metadata
I20260925 15:25:33.023957 8248 master_runner.cc:315] Copying system tablet from 127.1.163.126:36199
I20260925 15:25:33.024653 8248 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: tablet copy: Beginning tablet copy session from remote peer at address 127.1.163.126:36199
I20260925 15:25:33.031517 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.025049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.1.163.126:36199)
0925 15:25:33.025101 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.025104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.025212 (+ 108us) server_negotiation.cc:407] Connection header received
0925 15:25:33.025275 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.025276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.025285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.025330 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.025683 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.026164 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.027263 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.027473 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.027539 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.028642 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.028644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.028644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.028652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.030159 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.030578 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.030579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.030580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.030635 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.030881 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.030882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.030883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.031038 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.031136 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.031335 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.031414 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:25:33.031569 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.024947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58704)
0925 15:25:33.025126 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.025137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.025209 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.025345 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.025346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.025544 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.025613 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.025615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.026246 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.026247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.027143 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.027146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.027556 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.027557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.027616 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.028093 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.028099 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.028507 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.030264 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.030265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.030265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.030496 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.030703 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.030704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.030704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.030768 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.031165 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.031166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.031220 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.031303 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.031417 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":87,"threads_started":1}
I20260925 15:25:33.031915 7804 tablet_copy_service.cc:140] P c5bce094271e4f6e9cc745f750f9295f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3513a97b69314325830b8b2c75b0e6dd ({username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:58704)
I20260925 15:25:33.031999 7804 tablet_copy_service.cc:161] P c5bce094271e4f6e9cc745f750f9295f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3513a97b69314325830b8b2c75b0e6dd at {username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:58704: session id = 3513a97b69314325830b8b2c75b0e6dd-00000000000000000000000000000000
I20260925 15:25:33.033093 7804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:25:33.033751 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:25:33.034724 8248 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: tablet copy: Starting download of 0 data blocks...
I20260925 15:25:33.034843 8248 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: tablet copy: Starting download of 1 WAL segments...
I20260925 15:25:33.035794 8248 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:25:33.036711 7804 tablet_copy_service.cc:342] P c5bce094271e4f6e9cc745f750f9295f: Request end of tablet copy session 3513a97b69314325830b8b2c75b0e6dd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:58704
I20260925 15:25:33.036773 7804 tablet_copy_service.cc:433] P c5bce094271e4f6e9cc745f750f9295f: ending tablet copy session 3513a97b69314325830b8b2c75b0e6dd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3513a97b69314325830b8b2c75b0e6dd
I20260925 15:25:33.039287 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.040468 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.040588 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.040673 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.044797 8248 server_base.cc:1034] running on GCE node
I20260925 15:25:33.044941 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.045116 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.046245 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790349933046240 us; error 26 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.047607 8248 init.cc:377] Logged in from keytab as oryx/127.1.163.125@KRBTEST.COM (short username oryx)
I20260925 15:25:33.047904 8248 webserver.cc:460] Webserver started at http://127.1.163.125:39813/ using document root <none> and password file <none>
I20260925 15:25:33.048053 8248 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.048091 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.048774 8248 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:33.049386 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.049508 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:25:33.049556 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "3513a97b69314325830b8b2c75b0e6dd"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "43963ee8690416660427972a779b1009"
server_key_iv: "9b5cbef0773cdc7fea9dcf358d056b3c"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.049619 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.081422 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.083184 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.086522 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.125:45103 every 8 connection(s)
I20260925 15:25:33.086500 8248 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.125:45103
I20260925 15:25:33.086859 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
I20260925 15:25:33.087805 8418 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:25:33.088354 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Bootstrap starting.
I20260925 15:25:33.091388 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:25:33.091634 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Bootstrap complete.
I20260925 15:25:33.091842 8418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } }
I20260925 15:25:33.091935 8418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3513a97b69314325830b8b2c75b0e6dd, State: Initialized, Role: LEARNER
I20260925 15:25:33.091972 8418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } }
I20260925 15:25:33.092190 8418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:33.092175 8421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } }
I20260925 15:25:33.092290 8421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: LEARNER
I20260925 15:25:33.093767 8432 catalog_manager.cc:1259] Loaded cluster ID: 6abbc51a20e748c7b893c5f4a1347ba6
I20260925 15:25:33.094038 8432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: loading cluster ID for follower catalog manager: success
I20260925 15:25:33.094890 8432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: acquiring CA information for follower catalog manager: success
I20260925 15:25:33.095186 8432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:25:33.139130 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.132176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.1.163.125:45103)
0925 15:25:33.132323 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.132325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.132334 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:33.132366 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.132366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.132376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.132454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.133180 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.133827 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.134610 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.134872 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.134948 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.136320 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.136321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.136321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.136328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.137980 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.138331 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.138332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.138332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.138391 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.138592 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.138593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.138594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.138743 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.138856 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.138976 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.139051 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.1.163.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.124.keytab.
Entry for principal oryx/127.1.163.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.124.keytab.
I20260925 15:25:33.164131 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:44389
--webserver_interface=127.1.163.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.124.keytab
--principal=oryx/127.1.163.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:33.251155 8436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:33.251389 8436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:33.251411 8436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:33.252526 8436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:33.252635 8436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:33.252651 8436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:33.252664 8436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:33.252676 8436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:33.254135 8436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.124.keytab
--principal=oryx/127.1.163.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.163.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:33.254328 8436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:33.254494 8436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.259346 8441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.259395 8444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.259480 8442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.259819 8436 server_base.cc:1034] running on GCE node
I20260925 15:25:33.260102 8436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.260414 8436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.261586 8436 hybrid_clock.cc:648] HybridClock initialized: now 1790349933261567 us; error 36 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.264287 8436 init.cc:377] Logged in from keytab as oryx/127.1.163.124@KRBTEST.COM (short username oryx)
I20260925 15:25:33.264799 8436 webserver.cc:460] Webserver started at http://127.1.163.124:35389/ using document root <none> and password file <none>
I20260925 15:25:33.265070 8436 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.265118 8436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.265246 8436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:33.266302 8436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/instance:
uuid: "6fb7049e31c641a8bb43d27ac20927b0"
format_stamp: "Formatted at 2026-09-25 15:25:33 on dist-test-slave-hvv5"
server_key: "f1a3ff2ed8b74dcca17450ca20d0d31a"
server_key_iv: "eb92d0c94f3d59e580f0a12cd9ec61b5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.266795 8436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal/instance:
uuid: "6fb7049e31c641a8bb43d27ac20927b0"
format_stamp: "Formatted at 2026-09-25 15:25:33 on dist-test-slave-hvv5"
server_key: "f1a3ff2ed8b74dcca17450ca20d0d31a"
server_key_iv: "eb92d0c94f3d59e580f0a12cd9ec61b5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.268409 8436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:33.269496 8451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.269701 8436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:33.269775 8436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "6fb7049e31c641a8bb43d27ac20927b0"
format_stamp: "Formatted at 2026-09-25 15:25:33 on dist-test-slave-hvv5"
server_key: "f1a3ff2ed8b74dcca17450ca20d0d31a"
server_key_iv: "eb92d0c94f3d59e580f0a12cd9ec61b5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.269882 8436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.287563 8436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.290283 8436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:33.290511 8436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.301939 8436 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.124:44389
I20260925 15:25:33.302037 8503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.124:44389 every 8 connection(s)
I20260925 15:25:33.302517 8436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
I20260925 15:25:33.303577 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:33.305480 8504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:25:33.309672 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8436
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.124@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
I20260925 15:25:33.316146 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.306394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.1.163.126:36199)
0925 15:25:33.306475 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.306478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.306650 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:25:33.307244 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.307246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.307257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.307328 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.307960 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.308478 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.309834 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.310118 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.310197 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.313051 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.313053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.313054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.313063 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.314718 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.315163 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.315164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.315164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.315228 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.315511 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.315512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.315513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.315635 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.315670 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.315973 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.316056 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:25:33.316354 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.306205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58710)
0925 15:25:33.306510 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.306546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.307142 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.307390 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.307393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.307655 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.307869 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.307873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.308625 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.308626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.309467 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.309469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.310250 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.310252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.310471 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.311093 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.311102 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.312939 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.314829 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.314835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.314837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.315073 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.315297 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.315298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.315299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.315412 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.315707 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.315712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.315829 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.315939 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.316095 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
I20260925 15:25:33.316983 8504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:25:33.319229 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.310267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.1.163.124:44389)
0925 15:25:33.310559 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.310563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.310579 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:25:33.310596 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.310597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.310616 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.310709 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.311258 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.312161 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.313086 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.313520 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.313612 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.315860 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.315868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.315868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.315875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.318061 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.318445 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.318448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.318448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.318506 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.318756 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.318756 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.318757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.318884 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.318985 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.319105 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.319161 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":219,"threads_started":1}
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.124@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
I20260925 15:25:33.324455 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.317262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:51980)
0925 15:25:33.317342 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.317352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.317391 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.317542 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.317543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.317655 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.317701 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.317702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.318613 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.318614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.319434 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.319435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.320145 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.320146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.320206 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.320624 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.320628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.321843 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.323301 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.323302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.323302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.323455 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.323685 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.323686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.323686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.323739 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.324156 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.324157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.324208 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.324303 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.324380 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:25:33.324517 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.317341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.1.163.125:45103)
0925 15:25:33.317409 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.317412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.317422 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:33.317451 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.317453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.317467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.317530 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.317840 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.318507 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.319884 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.320077 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.320132 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.321926 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.321927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.321928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.321941 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.323221 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.323545 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.323546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.323547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.323611 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.323829 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.323830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.323830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.323961 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.324163 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.324371 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.324452 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:25:33.325018 8504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.124@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
I20260925 15:25:33.331683 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.325315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47176)
0925 15:25:33.325364 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.325375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.325445 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.325612 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.325613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.325730 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.325762 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.325764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.326362 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.326363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.326878 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.326880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.327287 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.327289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.327334 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.327835 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.327840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.329156 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.330604 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.330604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.330605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.330763 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.331069 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.331070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.331070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.331164 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.331420 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.331421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.331458 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.331514 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.331595 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:25:33.331736 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.325481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.1.163.124:44389)
0925 15:25:33.325516 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.325518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.325524 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:25:33.325540 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.325541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.325550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.325599 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.325828 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.326293 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.326948 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.327175 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.327233 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.329238 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.329239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.329239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.329246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.330516 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.330926 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.330927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.330928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.330996 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.331213 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.331213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.331214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.331361 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.331416 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.331531 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.331643 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":18}
I20260925 15:25:33.333321 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Bootstrap starting.
I20260925 15:25:33.334182 8504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:33.334465 8504 log.cc:824] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:33.335301 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: No bootstrap required, opened a new log
I20260925 15:25:33.336957 8504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } }
I20260925 15:25:33.337108 8504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:33.337155 8504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb7049e31c641a8bb43d27ac20927b0, State: Initialized, Role: FOLLOWER
I20260925 15:25:33.337309 8504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } }
I20260925 15:25:33.337626 8504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:33.337607 8509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } } }
I20260925 15:25:33.337745 8509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:25:33.339551 8520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:25:33.339720 8520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:25:33.342161 8436 master_runner.cc:428] Detected that this master 6fb7049e31c641a8bb43d27ac20927b0 is joining an existing cluster
I20260925 15:25:33.342211 8436 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.163.124:44389
I20260925 15:25:33.350679 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47180)
0925 15:25:33.343928 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.343937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.343994 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.344159 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.344160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.344388 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.344446 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.344447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345390 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.345391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.346343 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.346345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.346868 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.346924 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.347440 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.347445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.347783 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.349506 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.349507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.349507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.349719 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.349945 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.349946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.349947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.350022 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.350245 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.350246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.350341 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.350431 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.350538 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":32,"thread_start_us":195,"threads_started":1}
I20260925 15:25:33.350689 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.1.163.124:44389)
0925 15:25:33.343878 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.343880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.343975 (+ 95us) server_negotiation.cc:407] Connection header received
0925 15:25:33.344069 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.344070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.344078 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.344105 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.344520 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345299 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346441 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.346728 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346766 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.347908 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.347909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.347909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.347915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.349418 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.349819 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.349821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.349821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.349878 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.350071 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.350072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.350072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.350179 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.350255 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.350455 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.350608 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":37}
I20260925 15:25:33.365001 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:51994)
0925 15:25:33.343600 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.343609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.343649 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.344030 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.344031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.344323 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.344394 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.344396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345390 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.345391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.346343 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.346345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.346882 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.346947 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.361208 (+ 14261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.361216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.361675 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.363680 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.363681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.363682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.363932 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.364187 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.364188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.364250 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.364659 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.364660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.364710 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.364782 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.364907 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":36,"thread_start_us":86,"threads_started":1}
I20260925 15:25:33.365024 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.1.163.125:45103)
0925 15:25:33.343869 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.343871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.343880 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:33.343912 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.343913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.343924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.343981 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.344493 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345301 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346456 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.346760 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346797 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.361812 (+ 15015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.361814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.361814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.361824 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.363566 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364044 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.364045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.364046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.364116 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364376 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.364377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.364377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.364585 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.364670 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.364825 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.364923 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:25:33.365604 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.1.163.126:36199)
0925 15:25:33.343353 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.343355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.343458 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:25:33.343524 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.343525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.343536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.343596 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.344749 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345521 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346640 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.346944 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.346987 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.362193 (+ 15206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.362195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.362196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.362206 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.363875 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364316 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.364318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.364318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.364380 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364585 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.364586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.364586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.364723 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.364796 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.365407 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.365502 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:25:33.366083 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.343231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58724)
0925 15:25:33.343380 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.343392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.343454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.343646 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.343648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.344287 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.344638 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.344642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.345625 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.345626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.346534 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.346537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.347066 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.347067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.347196 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.361385 (+ 14189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.361391 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.362001 (+ 610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.363978 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.363979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.363979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.364227 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.364433 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.364433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.364434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.364487 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.365189 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.365190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.365252 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.365742 (+ 490us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.365868 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260925 15:25:33.370568 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7846
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349932, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.126@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
I20260925 15:25:33.377820 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:35985
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:36199
--webserver_port=41281
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:25:33.379637 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.368205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47188)
0925 15:25:33.368270 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.368282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.368333 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.368468 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.368468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.368679 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.368731 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.368732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.369374 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.369376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.370321 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.370329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.373100 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.373102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.373196 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.373732 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.373739 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.375572 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.377726 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.377728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.377729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.377961 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.378552 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.378553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.378553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.378620 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.378919 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.378920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.379223 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.379317 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.379487 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:25:33.380054 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.368263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.1.163.124:44389)
0925 15:25:33.368299 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.368301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.368315 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:25:33.368399 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.368399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.368407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.368450 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.368824 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.369293 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.370478 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.370739 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.370799 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.375707 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.375709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.375709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.375718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.377597 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.378388 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.378390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.378390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.378471 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.378683 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.378684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.378685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.378835 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.378893 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.379859 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.379965 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:25:33.380597 7784 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.163.124:44389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0"
member_type: NON_VOTER
last_known_addr {
host: "127.1.163.124"
port: 44389
}
attrs {
promote: true
}
}
dest_uuid: "c5bce094271e4f6e9cc745f750f9295f"
cas_config_opid_index: 5
I20260925 15:25:33.380895 7784 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } }
I20260925 15:25:33.381462 8344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.382514 8344 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.163.124:44389
I20260925 15:25:33.382583 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.382647 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.383050 8436 master.cc:537] Master@127.1.163.124:44389 shutting down...
I20260925 15:25:33.383114 8478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:33.384178 8478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:25:33.385253 8343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:33.385978 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.381547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.1.163.125:45103)
0925 15:25:33.381589 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.381591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.381660 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:25:33.381752 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.381753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.381777 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.381819 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:33.382032 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.382852 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.384495 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.385373 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.385433 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.385534 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.385828 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.385883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:25:33.386502 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.381445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:52010)
0925 15:25:33.381616 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.381626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.381683 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.381893 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.381894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.381896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:33.381943 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.381945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.382967 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.382968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.384350 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.384355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.385607 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.385609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.385687 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.385714 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.386329 (+ 615us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.386391 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
W20260925 15:25:33.386531 7769 consensus_peers.cc:487] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f -> Peer 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389): Couldn't send request to peer 6fb7049e31c641a8bb43d27ac20927b0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:25:33.386840 8392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 LEARNER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:25:33.387398 8343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:25:33.388073 8392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.388573 8436 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:33.388662 8436 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:33.388706 8436 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: stopping tablet replica
I20260925 15:25:33.389945 8343 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: attempting to promote NON_VOTER 3513a97b69314325830b8b2c75b0e6dd to VOTER
I20260925 15:25:33.389918 8421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bce094271e4f6e9cc745f750f9295f. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.390014 8421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: LEARNER
I20260925 15:25:33.390062 8421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: NON_VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: true } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.390120 8421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: LEARNER
I20260925 15:25:33.390146 8343 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } }
W20260925 15:25:33.390947 7769 connection.cc:535] client connection to 127.1.163.124:44389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:25:33.391832 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.391542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47202)
0925 15:25:33.391748 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.391801 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260925 15:25:33.391906 7769 consensus_peers.cc:487] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f -> Peer 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389): Couldn't send request to peer 6fb7049e31c641a8bb43d27ac20927b0. Status: Network error: Client connection negotiation failed: client connection to 127.1.163.124:44389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:25:33.392007 8392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 LEARNER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:25:33.392195 8532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:25:33.392894 8344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.393832 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.393915 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.393975 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.394016 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.394263 8392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.395195 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.395287 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:25:33.402999 8436 master.cc:552] Master@127.1.163.124:44389 shutdown complete.
I20260925 15:25:33.403064 8436 master_runner.cc:305] Clearing existing system tablet
I20260925 15:25:33.403213 8436 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.403263 8436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.403987 8436 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260925 15:25:33.404654 8534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.404804 8436 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.404871 8436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:33.404898 8436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "6fb7049e31c641a8bb43d27ac20927b0"
format_stamp: "Formatted at 2026-09-25 15:25:33 on dist-test-slave-hvv5"
server_key: "f1a3ff2ed8b74dcca17450ca20d0d31a"
server_key_iv: "eb92d0c94f3d59e580f0a12cd9ec61b5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.405670 8436 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:33.407397 8436 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:25:33.407471 8436 log.cc:1196] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:25:33.407673 8436 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Deleting consensus metadata
I20260925 15:25:33.407853 8436 master_runner.cc:315] Copying system tablet from 127.1.163.126:36199
I20260925 15:25:33.408584 8436 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: tablet copy: Beginning tablet copy session from remote peer at address 127.1.163.126:36199
I20260925 15:25:33.416000 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.408899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:58730)
0925 15:25:33.409022 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.409033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.409095 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.409259 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.409260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.409459 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.409534 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.409536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.410501 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.410503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.411319 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.411321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.411841 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.411842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.411912 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.412435 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.412441 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.412886 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.414753 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.414754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.414754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.414981 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.415222 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.415223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.415224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.415298 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.415606 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.415607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.415671 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.415765 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.415900 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260925 15:25:33.416046 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.408973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.1.163.126:36199)
0925 15:25:33.409021 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.409023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.409080 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:25:33.409152 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.409153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.409164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.409240 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.409618 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.410406 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.411411 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.411710 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.411779 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.413015 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.413017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.413017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.413026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.414656 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.415069 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.415071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.415071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.415145 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.415368 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.415369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.415369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.415520 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.415605 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.415876 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.415960 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:33.416244 7804 tablet_copy_service.cc:140] P c5bce094271e4f6e9cc745f750f9295f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fb7049e31c641a8bb43d27ac20927b0 ({username='oryx', principal='oryx/127.1.163.124@KRBTEST.COM'} at 127.0.0.1:58730)
I20260925 15:25:33.416308 7804 tablet_copy_service.cc:161] P c5bce094271e4f6e9cc745f750f9295f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fb7049e31c641a8bb43d27ac20927b0 at {username='oryx', principal='oryx/127.1.163.124@KRBTEST.COM'} at 127.0.0.1:58730: session id = 6fb7049e31c641a8bb43d27ac20927b0-00000000000000000000000000000000
I20260925 15:25:33.416947 7804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:25:33.417429 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:25:33.418757 8436 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: tablet copy: Starting download of 0 data blocks...
I20260925 15:25:33.418907 8436 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: tablet copy: Starting download of 1 WAL segments...
I20260925 15:25:33.419715 8436 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:25:33.420631 7804 tablet_copy_service.cc:342] P c5bce094271e4f6e9cc745f750f9295f: Request end of tablet copy session 6fb7049e31c641a8bb43d27ac20927b0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.163.124@KRBTEST.COM'} at 127.0.0.1:58730
I20260925 15:25:33.420696 7804 tablet_copy_service.cc:433] P c5bce094271e4f6e9cc745f750f9295f: ending tablet copy session 6fb7049e31c641a8bb43d27ac20927b0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fb7049e31c641a8bb43d27ac20927b0
I20260925 15:25:33.423694 8436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.425055 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.425644 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.425804 8436 server_base.cc:1034] running on GCE node
W20260925 15:25:33.425766 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.425956 8436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.426090 8436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.427223 8436 hybrid_clock.cc:648] HybridClock initialized: now 1790349933427231 us; error 45 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.428920 8436 init.cc:377] Logged in from keytab as oryx/127.1.163.124@KRBTEST.COM (short username oryx)
I20260925 15:25:33.429212 8436 webserver.cc:460] Webserver started at http://127.1.163.124:36051/ using document root <none> and password file <none>
I20260925 15:25:33.429337 8436 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.429371 8436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.430127 8436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:33.430785 8551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.430967 8436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:33.431025 8436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "6fb7049e31c641a8bb43d27ac20927b0"
format_stamp: "Formatted at 2026-09-25 15:25:33 on dist-test-slave-hvv5"
server_key: "f1a3ff2ed8b74dcca17450ca20d0d31a"
server_key_iv: "eb92d0c94f3d59e580f0a12cd9ec61b5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.431083 8436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.450815 8436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.452919 8436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.456096 8436 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.124:44389
I20260925 15:25:33.456211 8603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.124:44389 every 8 connection(s)
I20260925 15:25:33.456735 8436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
I20260925 15:25:33.457829 8604 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:25:33.458352 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Bootstrap starting.
I20260925 15:25:33.461912 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:25:33.462204 8604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: Bootstrap complete.
I20260925 15:25:33.462476 8604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } }
I20260925 15:25:33.462589 8604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fb7049e31c641a8bb43d27ac20927b0, State: Initialized, Role: LEARNER
I20260925 15:25:33.462634 8604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } }
I20260925 15:25:33.462863 8604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:33.462881 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.463101 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:25:33.465432 8618 catalog_manager.cc:1259] Loaded cluster ID: 6abbc51a20e748c7b893c5f4a1347ba6
I20260925 15:25:33.465528 8618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: loading cluster ID for follower catalog manager: success
I20260925 15:25:33.466660 8618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: acquiring CA information for follower catalog manager: success
I20260925 15:25:33.467031 8618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:25:33.481859 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:33.482088 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:33.482121 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:33.483408 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:33.483455 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:33.483485 8529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:33.485131 8529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=41281
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:33.485404 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:33.485672 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.491094 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.491147 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.491257 8529 server_base.cc:1034] running on GCE node
W20260925 15:25:33.491361 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.491590 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.491887 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.493063 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790349933493016 us; error 62 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.495841 8529 init.cc:377] Logged in from keytab as oryx/127.1.163.65@KRBTEST.COM (short username oryx)
I20260925 15:25:33.496294 8529 webserver.cc:460] Webserver started at http://127.1.163.65:41281/ using document root <none> and password file <none>
I20260925 15:25:33.496600 8529 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.496701 8529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.498212 8529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:33.498994 8633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.499230 8529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:33.499341 8529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "b336feea90e546059515bc83bffffcec"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "6bc72ac95bda6328497e7ea8b2b5590a"
server_key_iv: "e5fb7f63c9b1bdeb21704555e15fee99"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.499792 8529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.534485 8529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.536624 8529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:33.536792 8529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.537091 8529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:33.537426 8529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:33.537487 8529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.537532 8529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:33.537556 8529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.543754 8529 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35985
I20260925 15:25:33.543808 8746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35985 every 8 connection(s)
I20260925 15:25:33.544199 8529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:33.548530 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8529
I20260925 15:25:33.548744 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7980
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
I20260925 15:25:33.555608 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:39311
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:36199
--webserver_port=46005
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:25:33.557001 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42489 (local address 127.1.163.125:45103)
0925 15:25:33.545086 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.545089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.545335 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:25:33.546002 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.546003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.546016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.546049 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.547059 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547829 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549295 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.549626 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549668 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.553093 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.553095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.553095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.553104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.555414 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555800 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.555801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.555802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.555886 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.556198 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.556199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.556200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.556344 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.556389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.556812 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.556904 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:33.557001 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:45869 (local address 127.1.163.126:36199)
0925 15:25:33.545189 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.545191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.545524 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:25:33.546001 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.546003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.546016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.546049 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.547060 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547826 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549244 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.549541 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549585 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.553254 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.553256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.553256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.553265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.555029 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555802 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.555803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.555803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.555877 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.556198 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.556199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.556200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.556333 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.556389 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.556816 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.556903 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:33.557595 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.1.163.65:42489)
0925 15:25:33.545207 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.545249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.545874 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.546151 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.546156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.546716 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.546947 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.546953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547930 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.547931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.549197 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.549200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.549754 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.549893 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.550522 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.550533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.552976 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.555509 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.555511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.555708 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.555956 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.555957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.556115 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.556440 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.556448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.556647 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.557123 (+ 476us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.557385 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":251,"thread_start_us":90,"threads_started":1}
I20260925 15:25:33.557597 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:45869)
0925 15:25:33.545470 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.545478 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.545882 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.546161 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.546162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.546716 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.546947 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.546953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547894 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.547895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.549141 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.549144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.549749 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.549750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.549893 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.550567 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.550572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.553155 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.555116 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.555126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.555449 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.555956 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.555957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.555957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.556115 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.556440 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.556448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.556647 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.557098 (+ 451us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.557333 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":293,"thread_start_us":63,"threads_started":1}
I20260925 15:25:33.558065 8748 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:33.558197 8748 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.558369 8748 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:33.558113 8749 heartbeater.cc:347] Connected to a master server at 127.1.163.125:45103
I20260925 15:25:33.558529 8749 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.558588 8749 heartbeater.cc:511] Master 127.1.163.125:45103 requested a full tablet report, sending...
I20260925 15:25:33.558810 7784 ts_manager.cc:194] Re-registered known tserver with Master: b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
I20260925 15:25:33.558948 8382 ts_manager.cc:194] Registered new tserver with Master: b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
I20260925 15:25:33.559319 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:45869
I20260925 15:25:33.559693 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.1.163.65:43257)
0925 15:25:33.545325 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.545335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.545880 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.546441 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.546453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.546713 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.546946 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.546950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547824 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.547825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.549975 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.549977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.551045 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.551047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.551119 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.551552 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.551557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.553731 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.556779 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.556779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.556780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.557426 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.558727 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.558728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.558728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.558812 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.559307 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.559308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.559352 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.559413 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.559531 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":111,"thread_start_us":94,"threads_started":1}
I20260925 15:25:33.560114 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.545334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:43257 (local address 127.1.163.124:44389)
0925 15:25:33.545470 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.545472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.545481 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:33.545982 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.545983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.545997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.546073 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.547047 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.547715 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.550069 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.550922 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.550990 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.553845 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.553846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.553846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.553854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.556690 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.557516 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.557518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.557518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.557587 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.559042 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.559043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.559044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.559205 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.559255 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.559865 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.559973 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260925 15:25:33.560478 8747 heartbeater.cc:347] Connected to a master server at 127.1.163.124:44389
I20260925 15:25:33.560572 8747 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.560660 8747 heartbeater.cc:511] Master 127.1.163.124:44389 requested a full tablet report, sending...
I20260925 15:25:33.561077 8568 ts_manager.cc:194] Registered new tserver with Master: b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
W20260925 15:25:33.641098 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:33.641335 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:33.641360 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:33.642565 8755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:33.642606 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:33.642632 8755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:33.644093 8755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:39311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=46005
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:33.644292 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:33.644495 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.649731 8761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.649748 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.649937 8755 server_base.cc:1034] running on GCE node
W20260925 15:25:33.649951 8763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.650259 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.650532 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.651695 8755 hybrid_clock.cc:648] HybridClock initialized: now 1790349933651681 us; error 28 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.654538 8755 init.cc:377] Logged in from keytab as oryx/127.1.163.66@KRBTEST.COM (short username oryx)
I20260925 15:25:33.655015 8755 webserver.cc:460] Webserver started at http://127.1.163.66:46005/ using document root <none> and password file <none>
I20260925 15:25:33.655224 8755 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.655292 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.656772 8755 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:33.657349 8770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.657550 8755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:33.657634 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "85bb367b5ae8c6d02810f2c92a2bb958"
server_key_iv: "7a7faed0dde3d8861a60733518987048"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.657996 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.668321 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.670881 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:33.671047 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.671288 8755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:33.671645 8755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:33.671681 8755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.671710 8755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:33.671756 8755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.677659 8755 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39311
I20260925 15:25:33.677745 8883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39311 every 8 connection(s)
I20260925 15:25:33.678210 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:33.682593 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8755
I20260925 15:25:33.682763 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8114
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:33.688786 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:46707
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:36199
--webserver_port=37387
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
I20260925 15:25:33.692004 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.679254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:37909 (local address 127.1.163.126:36199)
0925 15:25:33.679299 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.679301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.680580 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 15:25:33.680615 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.680617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.680631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.680695 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.681550 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.682357 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.684764 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.685027 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.685127 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.687527 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.687529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.687530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.687538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.689433 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690849 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.690850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.690851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.690915 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.691237 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.691238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.691239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.691380 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.691419 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.691829 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.691910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:25:33.692045 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.679055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40187 (local address 127.1.163.124:44389)
0925 15:25:33.679106 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.679108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.680708 (+ 1600us) server_negotiation.cc:407] Connection header received
0925 15:25:33.680792 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.680793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.680805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.680855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.681535 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.682054 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.683918 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.684144 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.684233 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.687654 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.687656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.687656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.687664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.690238 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690858 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.690859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.690859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.690915 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.691237 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.691238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.691239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.691372 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.691419 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.691811 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.691913 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:33.692484 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.678942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:37909)
0925 15:25:33.680461 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.680468 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.680539 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.680717 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.680718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.681208 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.681429 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.681434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.682457 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.682457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.684671 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.684674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.685127 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.685128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.685196 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.685619 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.685624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.687409 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.690471 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.690472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.690753 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.690994 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.690995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.691155 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.691471 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.691478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.691654 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.692136 (+ 482us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.692243 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"mutex_wait_us":46,"thread_start_us":115,"threads_started":1}
I20260925 15:25:33.693136 8885 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:33.693241 8885 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.693449 8885 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:33.693719 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.678942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.1.163.66:40187)
0925 15:25:33.680650 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.680659 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.680733 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.680858 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.680858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.681187 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.681428 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.681432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.682163 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.682164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.683775 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.683777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.684243 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.684244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.684372 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.684990 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.685001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.687554 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.690340 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690350 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.690353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.690753 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.690994 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.690995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.690995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.691155 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.691471 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.691478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.691653 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.691788 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.691966 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"mutex_wait_us":66,"thread_start_us":85,"threads_started":1}
I20260925 15:25:33.693850 7784 ts_manager.cc:194] Re-registered known tserver with Master: bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311)
I20260925 15:25:33.694270 8884 heartbeater.cc:347] Connected to a master server at 127.1.163.124:44389
I20260925 15:25:33.694316 8884 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.694339 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:37909
I20260925 15:25:33.694379 8884 heartbeater.cc:511] Master 127.1.163.124:44389 requested a full tablet report, sending...
I20260925 15:25:33.694660 8568 ts_manager.cc:194] Registered new tserver with Master: bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311)
I20260925 15:25:33.695958 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.679240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:38157 (local address 127.1.163.125:45103)
0925 15:25:33.679288 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.679291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.679303 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:33.680429 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.680430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.680442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.680532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.681546 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.682321 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.689322 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.689530 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.689621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.691723 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.691725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.691725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.691733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.693885 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.695141 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.695142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.695143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.695200 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.695413 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.695414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.695431 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.695548 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.695627 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.695779 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.695861 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:33.696182 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.678942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.1.163.66:38157)
0925 15:25:33.679186 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.679229 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.680311 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.680634 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.680638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.681216 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.681429 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.681434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.688581 (+ 7147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.688582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.689219 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.689222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.689624 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.689625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.689692 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.690110 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.690116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.691631 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.694847 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.694848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.694848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.695056 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.695280 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.695281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.695282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.695339 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.695618 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.695619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.695661 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.695752 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.695862 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":6,"thread_start_us":108,"threads_started":1}
I20260925 15:25:33.696503 8886 heartbeater.cc:347] Connected to a master server at 127.1.163.125:45103
I20260925 15:25:33.696650 8886 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.696727 8886 heartbeater.cc:511] Master 127.1.163.125:45103 requested a full tablet report, sending...
I20260925 15:25:33.697038 8382 ts_manager.cc:194] Registered new tserver with Master: bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311)
W20260925 15:25:33.774865 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:33.775084 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:33.775108 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:33.776333 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:33.776381 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:33.776408 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:33.777988 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:46707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=37387
--tserver_master_addrs=127.1.163.126:36199,127.1.163.125:45103,127.1.163.124:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:33.778254 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:33.778525 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:33.783893 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.783746 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:33.783746 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:33.784289 8891 server_base.cc:1034] running on GCE node
I20260925 15:25:33.784500 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:33.784776 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:33.785923 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790349933785899 us; error 38 us; skew 500 ppm
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:33.788654 8891 init.cc:377] Logged in from keytab as oryx/127.1.163.67@KRBTEST.COM (short username oryx)
I20260925 15:25:33.789055 8891 webserver.cc:460] Webserver started at http://127.1.163.67:37387/ using document root <none> and password file <none>
I20260925 15:25:33.789243 8891 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:33.789296 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:33.790675 8891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:33.791226 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.791414 8891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:33.791507 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc"
format_stamp: "Formatted at 2026-09-25 15:25:32 on dist-test-slave-hvv5"
server_key: "dcd004cc9b77e505f23be3f0ba56cbac"
server_key_iv: "59db393926fea894058e11658155b7a5"
server_key_version: "encryptionkey@0"
I20260925 15:25:33.791899 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:33.826025 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:33.828183 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:33.828305 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:33.828516 8891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:33.828974 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:33.829011 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.829036 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:33.829094 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:33.835840 8891 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:46707
I20260925 15:25:33.835935 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:46707 every 8 connection(s)
I20260925 15:25:33.836314 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.125@KRBTEST.COM
Sep 25 15:25:33 dist-test-slave-hvv5 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349933, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.124@KRBTEST.COM
I20260925 15:25:33.846796 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 8891
I20260925 15:25:33.847589 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:47521 (local address 127.1.163.126:36199)
0925 15:25:33.837716 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.837755 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.837767 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:33.838156 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.838157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.838171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.838227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.839030 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.839552 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.840676 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.840887 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.840993 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.843785 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.843787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.843787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.843795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.845695 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.846318 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.846320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.846320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.846384 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.846712 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.846713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.846713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.846881 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.846931 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.847359 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.847467 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 15:25:33.848114 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34229 (local address 127.1.163.125:45103)
0925 15:25:33.837570 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.837572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.837584 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:33.838182 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.838183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.838197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.838244 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.839042 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.839896 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841092 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.841386 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841457 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.845136 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.845138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.845138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.845146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.846799 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847206 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.847207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.847207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.847260 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847451 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.847451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.847452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.847583 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.847627 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.847938 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.848016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260925 15:25:33.848148 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:56021 (local address 127.1.163.124:44389)
0925 15:25:33.837621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.837624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.837895 (+ 271us) server_negotiation.cc:407] Connection header received
0925 15:25:33.838238 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.838239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.838250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.838303 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.839041 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.839930 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841512 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.841740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841806 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.844807 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.844809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.844809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.844818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.846554 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847010 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.847011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.847012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.847072 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847280 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.847281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.847281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.847441 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.847487 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.847968 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.848056 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 15:25:33.848210 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:47521)
0925 15:25:33.837323 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.837368 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.838041 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.838392 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.838396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.838710 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.838927 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.838931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.839676 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.839677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.840516 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.840518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.841003 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.841163 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.841916 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.841928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.843674 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.845871 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.845881 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.845884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.846214 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.846477 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.846478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.846478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.846626 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.846985 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.846993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.847177 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.847691 (+ 514us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.847910 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":125,"threads_started":1}
I20260925 15:25:33.848210 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.1.163.67:34229)
0925 15:25:33.837324 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.837368 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.838068 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.838392 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.838397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.838789 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.838927 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.838933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.840015 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.840016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.840979 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.840980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.841506 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.841592 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.842039 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.842044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.844727 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.846895 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.846896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.846896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.847121 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.847332 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.847333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.847381 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.847746 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.847747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.847816 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.847880 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.848006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260925 15:25:33.848969 9021 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36199
I20260925 15:25:33.849103 9022 heartbeater.cc:347] Connected to a master server at 127.1.163.125:45103
I20260925 15:25:33.849056 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.837111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.1.163.67:56021)
0925 15:25:33.837819 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.837828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.838167 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.838392 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.838397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.838794 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:33.838927 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.838933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.840015 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.840016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.840973 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.840975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.841820 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.841821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.841887 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.842639 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:33.842644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:33.844730 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:33.846641 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.846642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.846643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.846923 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.847147 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:33.847147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:33.847147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:33.847206 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:33.847537 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:33.847538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:33.847577 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.848898 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.848985 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":300,"thread_start_us":79,"threads_started":1}
I20260925 15:25:33.849149 9022 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.849082 9021 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.849363 9022 heartbeater.cc:511] Master 127.1.163.125:45103 requested a full tablet report, sending...
I20260925 15:25:33.849542 9020 heartbeater.cc:347] Connected to a master server at 127.1.163.124:44389
I20260925 15:25:33.849612 9020 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:33.849691 9020 heartbeater.cc:511] Master 127.1.163.124:44389 requested a full tablet report, sending...
I20260925 15:25:33.849362 9021 heartbeater.cc:511] Master 127.1.163.126:36199 requested a full tablet report, sending...
I20260925 15:25:33.850065 8568 ts_manager.cc:194] Registered new tserver with Master: b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:33.850126 8382 ts_manager.cc:194] Registered new tserver with Master: b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:33.850236 7784 ts_manager.cc:194] Re-registered known tserver with Master: b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:33.850665 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:47521
I20260925 15:25:33.856604 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.851804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47226)
0925 15:25:33.851872 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.851881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.851919 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.852227 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.852228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.852230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:33.852276 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.852278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.853273 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.853274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.854876 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.854880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.856307 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.856308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.856379 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.856383 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.856415 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.856485 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:25:33.857071 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.851840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.1.163.124:44389)
0925 15:25:33.852074 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.852076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.852083 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:33.852100 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.852101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.852126 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.852340 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:33.852376 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.853163 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.854969 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.856205 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.856796 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.856914 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.856934 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.856989 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":51,"threads_started":1}
I20260925 15:25:33.857304 8578 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 15:25:33.857579 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5bce094271e4f6e9cc745f750f9295f. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.857699 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:25:33.859762 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.849703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.1.163.124:44389)
0925 15:25:33.849992 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.849994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.851620 (+ 1626us) server_negotiation.cc:407] Connection header received
0925 15:25:33.852033 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.852034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.852045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.852097 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.852852 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.853503 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.854542 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.854830 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.854887 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.856461 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.856463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.856463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.856470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.858272 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.858664 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.858665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.858666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.858729 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.858980 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.858982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.858982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.859126 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.859226 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.859595 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.859675 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260925 15:25:33.860198 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.849656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.1.163.126:36199)
0925 15:25:33.849743 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.849745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.850139 (+ 394us) server_negotiation.cc:407] Connection header received
0925 15:25:33.850190 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.850191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.850202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.850269 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.851559 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.852272 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.853453 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.853745 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.853817 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.856020 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.856022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.856022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.856031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.858688 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.859141 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.859142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.859143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.859218 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.859404 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.859405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.859406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.859535 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.859625 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.860005 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.860085 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:25:33.860249 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.849892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.1.163.125:45103)
0925 15:25:33.850851 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.850853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.850862 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:33.850881 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.850882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.850892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.850945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:33.851485 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.852247 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.853459 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.853748 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.853788 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.856650 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:33.856652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:33.856652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:33.856662 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:33.858597 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.859081 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.859082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.859082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.859137 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:33.859356 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:33.859358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:33.859358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:33.859502 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:33.859600 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.859808 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.860169 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933}
I20260925 15:25:33.861975 7784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:33.863912 7784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:33.864720 8343 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: NON_VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: true } } }
I20260925 15:25:33.864822 8343 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.866499 8343 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: attempting to promote NON_VOTER 6fb7049e31c641a8bb43d27ac20927b0 to VOTER
I20260925 15:25:33.866623 8343 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:33.866972 8578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 LEARNER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 15:25:33.867118 8392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Refusing update from remote peer c5bce094271e4f6e9cc745f750f9295f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 15:25:33.867173 8532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 15:25:33.867296 8344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 15:25:33.867987 8344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.868165 8392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.868172 8578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869028 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869122 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.869182 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869258 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.869233 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869308 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:25:33.869302 8344 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869354 8344 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:33.869321 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5bce094271e4f6e9cc745f750f9295f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:33.869413 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:25:33.874310 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39311 (local address 127.0.0.1:57400)
0925 15:25:33.870115 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.870128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.870172 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.870818 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.870818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.870820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:33.870880 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.870882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.871824 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.871825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.873026 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.873029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873990 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.874059 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874066 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.874099 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.874157 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":12}
I20260925 15:25:33.874513 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.1.163.67:46707)
0925 15:25:33.870590 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.870596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.870609 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:25:33.870655 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.870657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.870676 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.870939 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:33.871003 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.872137 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873161 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873880 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.874239 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874338 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.874368 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.874419 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":71,"threads_started":1}
I20260925 15:25:33.874531 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:46707 (local address 127.0.0.1:37792)
0925 15:25:33.870292 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.870300 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.870335 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.870783 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.870784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.870787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:33.870859 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.870862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.872197 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.872198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.873038 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.873040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873997 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.874059 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874075 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.874107 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.874157 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":12,"thread_start_us":103,"threads_started":1}
I20260925 15:25:33.874701 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35985 (local address 127.0.0.1:50556)
0925 15:25:33.870414 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:33.870422 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:33.870470 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:33.870851 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:33.870851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:33.870853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:33.870895 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.870896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.872148 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.872148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.873230 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:33.873232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873939 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:33.874000 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874073 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 15:25:33.874121 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:25:33.874194 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260925 15:25:33.874950 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.1.163.65:35985)
0925 15:25:33.870666 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.870670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.870703 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:25:33.870740 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.870743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.870758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.870978 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:33.871064 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.872066 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873320 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873833 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873935 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874323 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.874357 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.874869 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260925 15:25:33.875054 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:33.870415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.1.163.66:39311)
0925 15:25:33.870627 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:33.870631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:33.870642 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:33.870682 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:33.870689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:33.870725 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:33.870819 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:33.870938 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.871711 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.873117 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:33.873887 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:33.874234 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:33.874908 (+ 674us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:33.874934 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:25:33.874979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260925 15:25:33.875286 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 8441bb56809b4800a241dd4104664c9d (DEFAULT_TABLE table=test-table [id=fe886a9b8d3f4449b927c7ebf7d8a842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:33.875603 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8441bb56809b4800a241dd4104664c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:33.875663 8818 tablet_service.cc:1467] Processing CreateTablet for tablet 8441bb56809b4800a241dd4104664c9d (DEFAULT_TABLE table=test-table [id=fe886a9b8d3f4449b927c7ebf7d8a842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:33.875658 8681 tablet_service.cc:1467] Processing CreateTablet for tablet 8441bb56809b4800a241dd4104664c9d (DEFAULT_TABLE table=test-table [id=fe886a9b8d3f4449b927c7ebf7d8a842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:33.875981 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8441bb56809b4800a241dd4104664c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:33.875981 8681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8441bb56809b4800a241dd4104664c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:33.878522 9048 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Bootstrap starting.
I20260925 15:25:33.879225 9048 tablet_bootstrap.cc:654] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:33.879463 9050 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Bootstrap starting.
I20260925 15:25:33.879715 9048 log.cc:824] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:33.880129 9049 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Bootstrap starting.
I20260925 15:25:33.880234 9050 tablet_bootstrap.cc:654] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:33.880510 9048 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: No bootstrap required, opened a new log
I20260925 15:25:33.880618 9048 ts_tablet_manager.cc:1397] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:33.880616 9050 log.cc:824] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:33.880848 9049 tablet_bootstrap.cc:654] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:33.881201 9049 log.cc:824] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:33.881412 9050 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: No bootstrap required, opened a new log
I20260925 15:25:33.881513 9050 ts_tablet_manager.cc:1397] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:25:33.882025 9049 tablet_bootstrap.cc:492] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: No bootstrap required, opened a new log
I20260925 15:25:33.882145 9049 ts_tablet_manager.cc:1397] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:25:33.883452 9048 raft_consensus.cc:348] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.883641 9048 raft_consensus.cc:374] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:33.883567 9050 raft_consensus.cc:348] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.883708 9048 raft_consensus.cc:729] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a4e138ca2b452c8b1f82e5c6cf36fc, State: Initialized, Role: FOLLOWER
I20260925 15:25:33.883749 9050 raft_consensus.cc:374] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:33.883800 9050 raft_consensus.cc:729] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b336feea90e546059515bc83bffffcec, State: Initialized, Role: FOLLOWER
I20260925 15:25:33.883872 9048 consensus_queue.cc:260] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.883922 9050 consensus_queue.cc:260] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.884160 9048 ts_tablet_manager.cc:1428] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:33.884227 9050 ts_tablet_manager.cc:1428] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:25:33.884275 9021 heartbeater.cc:503] Master 127.1.163.126:36199 was elected leader, sending a full tablet report...
I20260925 15:25:33.884372 8748 heartbeater.cc:503] Master 127.1.163.126:36199 was elected leader, sending a full tablet report...
I20260925 15:25:33.886020 9049 raft_consensus.cc:348] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.886178 9049 raft_consensus.cc:374] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:33.886224 9049 raft_consensus.cc:729] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac9df4f3fcf43c4afe9c113d9b0d79b, State: Initialized, Role: FOLLOWER
I20260925 15:25:33.886339 9049 consensus_queue.cc:260] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:33.886818 9049 ts_tablet_manager.cc:1428] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 15:25:33.886907 8885 heartbeater.cc:503] Master 127.1.163.126:36199 was elected leader, sending a full tablet report...
W20260925 15:25:33.929687 8887 tablet.cc:2369] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:34.038872 9055 raft_consensus.cc:482] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:34.039004 9055 raft_consensus.cc:504] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:34.039357 9055 leader_election.cc:288] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311), b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:34.041906 9056 raft_consensus.cc:482] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:34.042037 9056 raft_consensus.cc:504] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:34.042415 9056 leader_election.cc:288] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b336feea90e546059515bc83bffffcec (127.1.163.65:35985), b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:34.043522 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.039579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:46707 (local address 127.1.163.65:46103)
0925 15:25:34.039645 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:34.039659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:34.039728 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:34.039995 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:34.039996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:34.039999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:34.040377 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.040378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.041294 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.041295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.042388 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.042391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.043223 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.043224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.043343 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.043347 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:34.043398 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:25:34.043439 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260925 15:25:34.043570 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.039781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42531 (local address 127.1.163.66:39311)
0925 15:25:34.039834 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.039836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.039847 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:34.039877 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.039878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.039887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.039945 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:34.040260 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.041048 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.042492 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.043115 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.043174 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.043185 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.043472 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.043511 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:25:34.043789 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.039784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:46103 (local address 127.1.163.67:46707)
0925 15:25:34.039840 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.039842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.039850 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:34.039877 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.039878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.039887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.039945 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:34.040481 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.041220 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.042532 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.043114 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.043179 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.043187 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.043687 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.043729 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:25:34.043973 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.039585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39311 (local address 127.1.163.65:42531)
0925 15:25:34.039704 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:34.039714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:34.039760 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:34.039995 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:34.039996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:34.039999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:34.040127 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.040129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.041133 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.041134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.042397 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.042398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.043214 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.043214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.043286 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.043300 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:25:34.043348 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:25:34.043392 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":139}
I20260925 15:25:34.044265 8838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "b336feea90e546059515bc83bffffcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" is_pre_election: true
I20260925 15:25:34.044399 8838 raft_consensus.cc:2454] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b336feea90e546059515bc83bffffcec in term 0.
I20260925 15:25:34.044402 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "b336feea90e546059515bc83bffffcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" is_pre_election: true
I20260925 15:25:34.044564 8974 raft_consensus.cc:2454] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b336feea90e546059515bc83bffffcec in term 0.
I20260925 15:25:34.044623 8637 leader_election.cc:302] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b336feea90e546059515bc83bffffcec, bac9df4f3fcf43c4afe9c113d9b0d79b; no voters:
I20260925 15:25:34.044744 9055 raft_consensus.cc:2781] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:34.044776 9055 raft_consensus.cc:482] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:34.044798 9055 raft_consensus.cc:3037] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:25:34.045722 8750 tablet.cc:2369] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:34.045711 9055 raft_consensus.cc:504] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:34.046078 8838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "b336feea90e546059515bc83bffffcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
I20260925 15:25:34.046165 8838 raft_consensus.cc:3037] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:34.046195 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "b336feea90e546059515bc83bffffcec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc"
I20260925 15:25:34.046276 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.042554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35985 (local address 127.1.163.66:52529)
0925 15:25:34.042655 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:34.042666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:34.042705 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:34.043017 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:34.043019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:34.043020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:34.043082 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.043083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.044272 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.044273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.045283 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.045286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.046035 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.046037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.046106 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.046110 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:34.046144 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 15:25:34.046187 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:25:34.046283 8974 raft_consensus.cc:3037] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:34.046557 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.042839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:59127 (local address 127.1.163.67:46707)
0925 15:25:34.043023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.043025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.043032 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:34.043053 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.043053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.043063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.043301 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:34.043342 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.044020 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.045384 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.046031 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.046338 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.046348 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.046367 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.046401 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260925 15:25:34.046803 9055 leader_election.cc:288] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [CANDIDATE]: Term 1 election: Requested vote from peers bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311), b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707)
I20260925 15:25:34.046849 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.042588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:46707 (local address 127.1.163.66:59127)
0925 15:25:34.042695 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:34.042705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:34.042743 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:34.043168 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:34.043168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:34.043170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:34.043235 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.043237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.044124 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.044126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.045286 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:34.045289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.046122 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.046123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:34.046209 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.046213 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:34.046246 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 15:25:34.046289 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1}
I20260925 15:25:34.047036 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.042793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:52529 (local address 127.1.163.65:35985)
0925 15:25:34.042901 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.042903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.042910 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:34.042931 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.042931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.042939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.043154 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:34.043198 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.044181 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.045385 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.045935 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.046914 (+ 979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.046924 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.046944 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.046983 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 15:25:34.047250 8838 raft_consensus.cc:2454] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b336feea90e546059515bc83bffffcec in term 1.
I20260925 15:25:34.047259 8974 raft_consensus.cc:2454] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b336feea90e546059515bc83bffffcec in term 1.
I20260925 15:25:34.047258 8701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b336feea90e546059515bc83bffffcec" is_pre_election: true
I20260925 15:25:34.047376 8701 raft_consensus.cc:2379] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bac9df4f3fcf43c4afe9c113d9b0d79b in current term 1: Already voted for candidate b336feea90e546059515bc83bffffcec in this term.
I20260925 15:25:34.047375 8973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8441bb56809b4800a241dd4104664c9d" candidate_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" is_pre_election: true
I20260925 15:25:34.047430 8634 leader_election.cc:302] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b336feea90e546059515bc83bffffcec, b9a4e138ca2b452c8b1f82e5c6cf36fc; no voters:
I20260925 15:25:34.047457 8973 raft_consensus.cc:2379] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bac9df4f3fcf43c4afe9c113d9b0d79b in current term 1: Already voted for candidate b336feea90e546059515bc83bffffcec in this term.
I20260925 15:25:34.047514 9055 raft_consensus.cc:2781] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:34.047655 9055 raft_consensus.cc:686] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 LEADER]: Becoming Leader. State: Replica: b336feea90e546059515bc83bffffcec, State: Running, Role: LEADER
I20260925 15:25:34.047681 8774 leader_election.cc:302] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bac9df4f3fcf43c4afe9c113d9b0d79b; no voters: b336feea90e546059515bc83bffffcec, b9a4e138ca2b452c8b1f82e5c6cf36fc
I20260925 15:25:34.047749 9055 consensus_queue.cc:237] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } }
I20260925 15:25:34.047848 9056 raft_consensus.cc:2726] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:25:34.048599 7784 catalog_manager.cc:5495] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec reported cstate change: term changed from 0 to 1, leader changed from <none> to b336feea90e546059515bc83bffffcec (127.1.163.65). New cstate: current_term: 1 leader_uuid: "b336feea90e546059515bc83bffffcec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:34.057519 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.054915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.1.163.65:35985)
0925 15:25:34.054971 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.054973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.054984 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:34.055079 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.055079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.055096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.055170 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:34.055318 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.055856 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.056732 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.056926 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.057002 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.057183 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:34.057282 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:34.057366 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.057407 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.057431 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:25:34.059317 8838 raft_consensus.cc:1261] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Refusing update from remote peer b336feea90e546059515bc83bffffcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:34.059317 8973 raft_consensus.cc:1261] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Refusing update from remote peer b336feea90e546059515bc83bffffcec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:34.059679 9058 consensus_queue.cc:1035] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:34.059860 9055 consensus_queue.cc:1035] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:34.062610 9063 mvcc.cc:205] Tried to move back new op lower bound from 7333273329903812608 to 7333273329860263936. Current Snapshot: MvccSnapshot[applied={T|T < 7333273329903812608}]
I20260925 15:25:34.064754 9062 mvcc.cc:205] Tried to move back new op lower bound from 7333273329903812608 to 7333273329860263936. Current Snapshot: MvccSnapshot[applied={T|T < 7333273329903812608}]
I20260925 15:25:34.066493 7783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:34.066589 7783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:34.067989 7783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5bce094271e4f6e9cc745f750f9295f: Sending DeleteTablet for 3 replicas of tablet 8441bb56809b4800a241dd4104664c9d
I20260925 15:25:34.068403 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet 8441bb56809b4800a241dd4104664c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:34 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:37792
I20260925 15:25:34.068436 8818 tablet_service.cc:1514] Processing DeleteTablet for tablet 8441bb56809b4800a241dd4104664c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:34 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:57400
I20260925 15:25:34.068640 9071 tablet_replica.cc:330] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: stopping tablet replica
I20260925 15:25:34.068737 9071 raft_consensus.cc:2229] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:34.068833 9071 raft_consensus.cc:2258] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:34.069078 8681 tablet_service.cc:1514] Processing DeleteTablet for tablet 8441bb56809b4800a241dd4104664c9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:34 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50556
I20260925 15:25:34.069562 9071 ts_tablet_manager.cc:1905] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:34.069574 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 7752
I20260925 15:25:34.071477 9071 ts_tablet_manager.cc:1918] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:34.071552 9071 log.cc:1196] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/8441bb56809b4800a241dd4104664c9d
I20260925 15:25:34.071817 9071 ts_tablet_manager.cc:1939] T 8441bb56809b4800a241dd4104664c9d P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting consensus metadata
I20260925 15:25:34.073307 9072 tablet_replica.cc:330] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: stopping tablet replica
I20260925 15:25:34.073444 9072 raft_consensus.cc:2229] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:34.073572 9072 raft_consensus.cc:2258] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:34.074540 9072 ts_tablet_manager.cc:1905] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:34.076586 9072 ts_tablet_manager.cc:1918] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:34.076658 9072 log.cc:1196] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/8441bb56809b4800a241dd4104664c9d
W20260925 15:25:34.076851 8908 connection.cc:535] server connection from 127.0.0.1:37792 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:25:34.077762 9074 tablet_replica.cc:330] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: stopping tablet replica
I20260925 15:25:34.077884 9074 raft_consensus.cc:2229] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:34.078035 9074 raft_consensus.cc:2258] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:34.078466 9072 ts_tablet_manager.cc:1939] T 8441bb56809b4800a241dd4104664c9d P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting consensus metadata
W20260925 15:25:34.078905 8773 connection.cc:441] server connection from 127.0.0.1:57400 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57400 (request call id 1) could send its response
I20260925 15:25:34.079303 9074 ts_tablet_manager.cc:1905] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:34.081177 9074 ts_tablet_manager.cc:1918] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:34.081254 9074 log.cc:1196] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/8441bb56809b4800a241dd4104664c9d
I20260925 15:25:34.081570 9074 ts_tablet_manager.cc:1939] T 8441bb56809b4800a241dd4104664c9d P b336feea90e546059515bc83bffffcec: Deleting consensus metadata
W20260925 15:25:34.081952 8636 connection.cc:441] server connection from 127.0.0.1:50556 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50556 (request call id 1) could send its response
I20260925 15:25:34.082823 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.068969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.1.163.124:44389)
0925 15:25:34.069015 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.069018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.069624 (+ 606us) server_negotiation.cc:407] Connection header received
0925 15:25:34.070300 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.070301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.070313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.070354 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:34.071209 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.072043 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.073713 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.074070 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.074117 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.079637 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:34.079639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:34.079639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:34.079648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:34.081543 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.081984 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.081986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.081986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.082047 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.082275 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.082276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.082276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.082415 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:34.082527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.082635 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.082713 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:25:34.088881 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.078793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.1.163.124:44389)
0925 15:25:34.078854 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.078855 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.079582 (+ 727us) server_negotiation.cc:407] Connection header received
0925 15:25:34.079677 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.079678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.079690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.079720 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:34.080384 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.081189 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.083613 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.083895 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.083967 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.085504 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:34.085505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:34.085505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:34.085512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:34.087376 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.087820 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.087821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.087821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.087878 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.088222 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.088223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.088223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.088369 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:34.088460 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.088698 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.088787 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:25:34.089802 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:34.079149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.1.163.125:45103)
0925 15:25:34.079193 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:34.079195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:34.081764 (+ 2569us) server_negotiation.cc:407] Connection header received
0925 15:25:34.081865 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:34.081866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:34.081878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:34.081909 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:34.083678 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.084346 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.085158 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:34.085376 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:34.085484 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:34.086870 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:34.086871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:34.086872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:34.086880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:34.088578 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.088980 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.088981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.088981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.089035 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:34.089222 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:34.089222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:34.089223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:34.089337 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:34.089406 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:34.089629 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:25:34.089710 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:25:35.063766 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.063414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:41409)
0925 15:25:35.063605 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.063699 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260925 15:25:35.064045 8885 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.065531 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.065272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:43881)
0925 15:25:35.065398 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.065477 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260925 15:25:35.065671 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.065381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:58839)
0925 15:25:35.065558 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.065624 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
W20260925 15:25:35.065806 9021 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
W20260925 15:25:35.065943 8748 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36199 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.568845 9086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5bce094271e4f6e9cc745f750f9295f)
I20260925 15:25:35.568845 9085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5bce094271e4f6e9cc745f750f9295f)
I20260925 15:25:35.569015 9086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:35.569037 9085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:35.569340 9086 leader_election.cc:288] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389)
I20260925 15:25:35.569468 9085 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125:45103)
I20260925 15:25:35.569732 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59234)
0925 15:25:35.569633 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.569683 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":40,"threads_started":1}
I20260925 15:25:35.569820 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59226)
0925 15:25:35.569710 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.569771 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":50,"threads_started":1}
W20260925 15:25:35.569983 8554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
W20260925 15:25:35.570026 8368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.570468 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.570376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59250)
0925 15:25:35.570411 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.570436 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:25:35.570467 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.570371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59244)
0925 15:25:35.570404 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.570446 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 15:25:35.570608 8368 leader_election.cc:334] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
W20260925 15:25:35.570609 8554 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.572935 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.1.163.124:44389)
0925 15:25:35.569932 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:35.569933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:35.569941 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:35.569970 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:35.569971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:35.569981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:35.570063 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:35.570526 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.571068 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572019 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.572592 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:35.572682 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:35.572814 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:25:35.572860 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":52,"threads_started":1}
I20260925 15:25:35.572983 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:44389 (local address 127.0.0.1:47240)
0925 15:25:35.569789 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.569800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:35.569862 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:35.570120 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:35.570121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:35.570124 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:35.570203 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:35.570205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.571150 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.571151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:35.571912 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:35.571914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.572671 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:35.572735 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:35.572740 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:35.572786 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:25:35.572854 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":62,"threads_started":1}
I20260925 15:25:35.573297 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:45103 (local address 127.0.0.1:52038)
0925 15:25:35.570125 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.570134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:35.570179 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:35.570366 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:35.570367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:35.570369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:35.570441 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:35.570443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.571199 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.571200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:35.572083 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:35.572086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.572844 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:35.572903 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:35.572908 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:35.572968 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:25:35.573218 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":50,"threads_started":1}
I20260925 15:25:35.573318 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.569802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.1.163.125:45103)
0925 15:25:35.569887 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:35.569889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:35.570230 (+ 341us) server_negotiation.cc:407] Connection header received
0925 15:25:35.570256 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:35.570257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:35.570268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:35.570325 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:35.570525 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.571100 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572166 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:35.572745 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:35.572831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:35.572842 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:35.572982 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:25:35.573264 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":40,"threads_started":1}
I20260925 15:25:35.573434 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3513a97b69314325830b8b2c75b0e6dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb7049e31c641a8bb43d27ac20927b0" is_pre_election: true
I20260925 15:25:35.573549 8578 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3513a97b69314325830b8b2c75b0e6dd in term 1.
I20260925 15:25:35.573719 8367 leader_election.cc:302] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3513a97b69314325830b8b2c75b0e6dd, 6fb7049e31c641a8bb43d27ac20927b0; no voters: c5bce094271e4f6e9cc745f750f9295f
I20260925 15:25:35.573695 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb7049e31c641a8bb43d27ac20927b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3513a97b69314325830b8b2c75b0e6dd" is_pre_election: true
I20260925 15:25:35.573804 8392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb7049e31c641a8bb43d27ac20927b0 in term 1.
I20260925 15:25:35.573848 9086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:25:35.573884 9086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5bce094271e4f6e9cc745f750f9295f)
I20260925 15:25:35.573907 9086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:25:35.573971 8552 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3513a97b69314325830b8b2c75b0e6dd, 6fb7049e31c641a8bb43d27ac20927b0; no voters: c5bce094271e4f6e9cc745f750f9295f
I20260925 15:25:35.574066 9085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:25:35.574108 9085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5bce094271e4f6e9cc745f750f9295f)
I20260925 15:25:35.574141 9085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:25:35.574731 9086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:35.574818 9085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:35.574878 9086 leader_election.cc:288] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 election: Requested vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389)
I20260925 15:25:35.574970 9085 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 election: Requested vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 3513a97b69314325830b8b2c75b0e6dd (127.1.163.125:45103)
I20260925 15:25:35.575067 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3513a97b69314325830b8b2c75b0e6dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb7049e31c641a8bb43d27ac20927b0"
I20260925 15:25:35.575172 8578 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3513a97b69314325830b8b2c75b0e6dd in current term 2: Already voted for candidate 6fb7049e31c641a8bb43d27ac20927b0 in this term.
I20260925 15:25:35.575182 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.575044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59274)
0925 15:25:35.575133 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.575159 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:25:35.575198 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb7049e31c641a8bb43d27ac20927b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3513a97b69314325830b8b2c75b0e6dd"
I20260925 15:25:35.575299 8392 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fb7049e31c641a8bb43d27ac20927b0 in current term 2: Already voted for candidate 3513a97b69314325830b8b2c75b0e6dd in this term.
I20260925 15:25:35.575287 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.574967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59270)
0925 15:25:35.575010 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.575036 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:25:35.575697 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.575517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59288)
0925 15:25:35.575619 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.575667 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 15:25:35.575785 8554 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.575840 8554 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb7049e31c641a8bb43d27ac20927b0; no voters: 3513a97b69314325830b8b2c75b0e6dd, c5bce094271e4f6e9cc745f750f9295f
I20260925 15:25:35.575941 9085 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:25:35.576097 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:35.575738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59294)
0925 15:25:35.576041 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:35.576069 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260925 15:25:35.576193 8368 leader_election.cc:334] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:35.576246 8368 leader_election.cc:302] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3513a97b69314325830b8b2c75b0e6dd; no voters: 6fb7049e31c641a8bb43d27ac20927b0, c5bce094271e4f6e9cc745f750f9295f
I20260925 15:25:35.576326 9086 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:25:36.064754 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:36.064489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:33661)
0925 15:25:36.064654 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:36.064700 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
I20260925 15:25:36.066432 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:36.066160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:34889)
0925 15:25:36.066341 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:36.066380 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
I20260925 15:25:36.066699 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:36.066428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:53205)
0925 15:25:36.066623 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:36.066655 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260925 15:25:37.065506 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.065218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:54069)
0925 15:25:37.065388 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.065440 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260925 15:25:37.067238 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.066979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:56087)
0925 15:25:37.067149 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.067186 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
I20260925 15:25:37.067359 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.067148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:50093)
0925 15:25:37.067251 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.067296 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260925 15:25:37.309967 9101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:37.310117 9101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:37.310470 9101 leader_election.cc:288] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389)
I20260925 15:25:37.310804 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.310617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59518)
0925 15:25:37.310726 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.310768 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260925 15:25:37.310813 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3513a97b69314325830b8b2c75b0e6dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb7049e31c641a8bb43d27ac20927b0" is_pre_election: true
I20260925 15:25:37.310952 8578 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3513a97b69314325830b8b2c75b0e6dd in term 2.
I20260925 15:25:37.311128 8367 leader_election.cc:302] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3513a97b69314325830b8b2c75b0e6dd, 6fb7049e31c641a8bb43d27ac20927b0; no voters:
I20260925 15:25:37.311252 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.311135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59532)
0925 15:25:37.311185 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.311215 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:25:37.311264 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 15:25:37.311318 9101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:37.311344 9101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 2 FOLLOWER]: Advancing to term 3
W20260925 15:25:37.311358 8368 leader_election.cc:334] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:37.312076 9101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:37.312228 9101 leader_election.cc:288] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 election: Requested vote from peers c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199), 6fb7049e31c641a8bb43d27ac20927b0 (127.1.163.124:44389)
I20260925 15:25:37.312448 8578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3513a97b69314325830b8b2c75b0e6dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb7049e31c641a8bb43d27ac20927b0"
I20260925 15:25:37.312525 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.312408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59548)
0925 15:25:37.312466 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.312508 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:25:37.312579 8578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:25:37.313019 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.312895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59564)
0925 15:25:37.312951 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.312995 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:25:37.313113 8368 leader_election.cc:334] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
I20260925 15:25:37.313452 8578 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3513a97b69314325830b8b2c75b0e6dd in term 3.
I20260925 15:25:37.313618 8367 leader_election.cc:302] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3513a97b69314325830b8b2c75b0e6dd, 6fb7049e31c641a8bb43d27ac20927b0; no voters: c5bce094271e4f6e9cc745f750f9295f
I20260925 15:25:37.313707 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:25:37.313814 9101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [term 3 LEADER]: Becoming Leader. State: Replica: 3513a97b69314325830b8b2c75b0e6dd, State: Running, Role: LEADER
I20260925 15:25:37.313892 9101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } }
I20260925 15:25:37.314263 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3513a97b69314325830b8b2c75b0e6dd. Latest consensus state: current_term: 3 leader_uuid: "3513a97b69314325830b8b2c75b0e6dd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:37.314369 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:37.314549 9106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:37.314967 9106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:37.315026 9106 catalog_manager.cc:1259] Loaded cluster ID: 6abbc51a20e748c7b893c5f4a1347ba6
I20260925 15:25:37.315043 9106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:37.315187 9106 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:37.315239 9106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Loaded TSK: 0
I20260925 15:25:37.315338 9106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:37.339466 8381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:37.341327 8381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:37.342429 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.342278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59580)
0925 15:25:37.342345 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.342390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:25:37.342519 8578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [term 3 FOLLOWER]: Refusing update from remote peer 3513a97b69314325830b8b2c75b0e6dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 15:25:37.342943 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.342792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.0.0.1:59582)
0925 15:25:37.342850 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.342885 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:25:37.342891 9104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 15:25:37.343037 8368 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd -> Peer c5bce094271e4f6e9cc745f750f9295f (127.1.163.126:36199): Couldn't send request to peer c5bce094271e4f6e9cc745f750f9295f. Status: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:25:37.343607 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3513a97b69314325830b8b2c75b0e6dd. Latest consensus state: current_term: 3 leader_uuid: "3513a97b69314325830b8b2c75b0e6dd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:37.343725 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:25:37.344324 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3513a97b69314325830b8b2c75b0e6dd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:37.344429 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:37.344586 9108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3513a97b69314325830b8b2c75b0e6dd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c5bce094271e4f6e9cc745f750f9295f" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36199 } } peers { permanent_uuid: "3513a97b69314325830b8b2c75b0e6dd" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 45103 } attrs { promote: false } } peers { permanent_uuid: "6fb7049e31c641a8bb43d27ac20927b0" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 44389 } attrs { promote: false } } }
I20260925 15:25:37.344686 9108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb7049e31c641a8bb43d27ac20927b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:25:37.344795 9109 mvcc.cc:205] Tried to move back new op lower bound from 7333273343352373248 to 7333273343238459392. Current Snapshot: MvccSnapshot[applied={T|T < 7333273343352373248}]
I20260925 15:25:37.349678 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.1.163.66:39311)
0925 15:25:37.346533 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.346535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.346641 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:25:37.346674 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.346676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.346687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.346909 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:37.346949 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.347513 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.348750 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.349330 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.349411 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.349559 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.349605 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 15:25:37.349695 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39311 (local address 127.0.0.1:57416)
0925 15:25:37.346512 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.346523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:37.346582 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:37.346779 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:37.346780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:37.346781 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:37.346843 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.346845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.347602 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.347603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.348655 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.348657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.349406 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.349472 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.349477 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:37.349546 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:25:37.349592 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260925 15:25:37.349946 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:46707 (local address 127.0.0.1:37800)
0925 15:25:37.346732 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.346742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:37.346784 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:37.347028 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:37.347029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:37.347031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:37.347067 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.347068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.348034 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.348035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.348983 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.348986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.349730 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.349793 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.349798 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:37.349833 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 15:25:37.349872 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260925 15:25:37.350271 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35985 (local address 127.0.0.1:50582)
0925 15:25:37.346704 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.346716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:37.346779 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:37.347032 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:37.347033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:37.347035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:37.347085 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.347087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.348019 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.348019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.349166 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.349168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.350024 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.350025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.350085 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.350088 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:37.350133 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:25:37.350206 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":29,"thread_start_us":83,"threads_started":1}
I20260925 15:25:37.350304 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.1.163.67:46707)
0925 15:25:37.346884 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.346887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.346894 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:37.346923 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.346923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.346936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.347015 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:37.347147 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.347912 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349074 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.349616 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.350147 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.350165 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.350188 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.350231 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":54,"threads_started":1}
I20260925 15:25:37.350334 8818 tablet_service.cc:1467] Processing CreateTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 (DEFAULT_TABLE table=test-table [id=89172a76c912400e9b8024d863c475cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:37.350481 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097d762c5f5e4da2b5e81c60e0ccf842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:37.350517 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.346562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.1.163.65:35985)
0925 15:25:37.346668 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.346670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.346850 (+ 180us) server_negotiation.cc:407] Connection header received
0925 15:25:37.346888 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.346889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.346905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.346979 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:37.347176 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.347933 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349257 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.349915 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.349981 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.349993 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.350392 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.350452 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260925 15:25:37.351863 9119 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Bootstrap starting.
I20260925 15:25:37.351972 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 (DEFAULT_TABLE table=test-table [id=89172a76c912400e9b8024d863c475cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:37.352100 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097d762c5f5e4da2b5e81c60e0ccf842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:37.352255 8681 tablet_service.cc:1467] Processing CreateTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 (DEFAULT_TABLE table=test-table [id=89172a76c912400e9b8024d863c475cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:37.352490 8681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097d762c5f5e4da2b5e81c60e0ccf842. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:37.352602 9119 tablet_bootstrap.cc:654] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:37.353291 9119 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: No bootstrap required, opened a new log
I20260925 15:25:37.353351 9119 ts_tablet_manager.cc:1397] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:25:37.353477 9119 raft_consensus.cc:348] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.353566 9119 raft_consensus.cc:374] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:37.353588 9119 raft_consensus.cc:729] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac9df4f3fcf43c4afe9c113d9b0d79b, State: Initialized, Role: FOLLOWER
I20260925 15:25:37.353660 9119 consensus_queue.cc:260] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.353922 9119 ts_tablet_manager.cc:1428] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:37.354269 9123 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec: Bootstrap starting.
I20260925 15:25:37.354408 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.354296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:59161)
0925 15:25:37.354343 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.354373 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 15:25:37.354501 9121 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Bootstrap starting.
I20260925 15:25:37.354831 9123 tablet_bootstrap.cc:654] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:37.355000 9121 tablet_bootstrap.cc:654] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:37.355151 8886 heartbeater.cc:503] Master 127.1.163.125:45103 was elected leader, sending a full tablet report...
I20260925 15:25:37.355347 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.355261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:35561)
0925 15:25:37.355298 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.355324 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:25:37.355587 9123 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec: No bootstrap required, opened a new log
I20260925 15:25:37.355669 9123 ts_tablet_manager.cc:1397] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:37.355803 9123 raft_consensus.cc:348] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.355886 9123 raft_consensus.cc:374] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:37.355921 9123 raft_consensus.cc:729] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b336feea90e546059515bc83bffffcec, State: Initialized, Role: FOLLOWER
I20260925 15:25:37.355966 9123 consensus_queue.cc:260] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.356140 9123 ts_tablet_manager.cc:1428] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:37.356194 9121 tablet_bootstrap.cc:492] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: No bootstrap required, opened a new log
I20260925 15:25:37.356238 9121 ts_tablet_manager.cc:1397] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:37.356419 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.356321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:57409)
0925 15:25:37.356363 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.356389 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 15:25:37.356428 9121 raft_consensus.cc:348] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.356499 9121 raft_consensus.cc:374] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:37.356564 9121 raft_consensus.cc:729] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a4e138ca2b452c8b1f82e5c6cf36fc, State: Initialized, Role: FOLLOWER
I20260925 15:25:37.356643 9121 consensus_queue.cc:260] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.356734 8749 heartbeater.cc:503] Master 127.1.163.125:45103 was elected leader, sending a full tablet report...
I20260925 15:25:37.356837 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.356753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:45493)
0925 15:25:37.356790 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.356812 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:25:37.356866 9121 ts_tablet_manager.cc:1428] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:37.357306 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.357213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:51149)
0925 15:25:37.357251 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.357279 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:25:37.357627 9022 heartbeater.cc:503] Master 127.1.163.125:45103 was elected leader, sending a full tablet report...
I20260925 15:25:37.357703 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.357616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:55485)
0925 15:25:37.357653 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.357683 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 15:25:37.560500 9127 raft_consensus.cc:482] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:37.560662 9127 raft_consensus.cc:504] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.561031 9127 leader_election.cc:288] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311), b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
I20260925 15:25:37.564149 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.561233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39311 (local address 127.1.163.67:49655)
0925 15:25:37.561301 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.561313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:37.561371 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:37.561548 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:37.561550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:37.561552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:37.561617 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.561619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.562308 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.562309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.563257 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.563259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.563872 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.563873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.563937 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.563942 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:25:37.563996 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:25:37.564040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:25:37.564160 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.561301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:49655 (local address 127.1.163.66:39311)
0925 15:25:37.561365 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.561368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.561383 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:25:37.561423 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.561424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.561435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.561538 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:37.561692 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.562207 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.563346 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.563785 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.563871 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.563881 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.564019 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.564051 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:25:37.564438 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.561346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35985 (local address 127.1.163.67:57215)
0925 15:25:37.561564 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.561573 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:37.561626 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:37.561750 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:37.561751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:37.561752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:37.561856 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.561858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.562545 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.562546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.563362 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:37.563364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.564139 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.564140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:37.564212 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.564216 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:37.564275 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:25:37.564341 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":144,"threads_started":1}
I20260925 15:25:37.564427 8838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097d762c5f5e4da2b5e81c60e0ccf842" candidate_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" is_pre_election: true
I20260925 15:25:37.564519 8838 raft_consensus.cc:2454] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9a4e138ca2b452c8b1f82e5c6cf36fc in term 0.
I20260925 15:25:37.564743 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.561446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:57215 (local address 127.1.163.65:35985)
0925 15:25:37.561521 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.561523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.561615 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:25:37.561680 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.561681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.561691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.561746 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:37.561939 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.562460 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.563458 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.564037 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.564103 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.564113 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.564626 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.564681 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:25:37.564745 8910 leader_election.cc:302] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9a4e138ca2b452c8b1f82e5c6cf36fc, bac9df4f3fcf43c4afe9c113d9b0d79b; no voters:
I20260925 15:25:37.564879 9127 raft_consensus.cc:2781] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:37.564913 9127 raft_consensus.cc:482] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:37.564936 9127 raft_consensus.cc:3037] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:37.564966 8701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097d762c5f5e4da2b5e81c60e0ccf842" candidate_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b336feea90e546059515bc83bffffcec" is_pre_election: true
I20260925 15:25:37.565063 8701 raft_consensus.cc:2454] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9a4e138ca2b452c8b1f82e5c6cf36fc in term 0.
I20260925 15:25:37.565744 9127 raft_consensus.cc:504] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.565882 9127 leader_election.cc:288] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [CANDIDATE]: Term 1 election: Requested vote from peers bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311), b336feea90e546059515bc83bffffcec (127.1.163.65:35985)
I20260925 15:25:37.566051 8838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097d762c5f5e4da2b5e81c60e0ccf842" candidate_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b"
I20260925 15:25:37.566071 8701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097d762c5f5e4da2b5e81c60e0ccf842" candidate_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b336feea90e546059515bc83bffffcec"
I20260925 15:25:37.566129 8838 raft_consensus.cc:3037] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:37.566138 8701 raft_consensus.cc:3037] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:37.566890 8838 raft_consensus.cc:2454] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9a4e138ca2b452c8b1f82e5c6cf36fc in term 1.
I20260925 15:25:37.566913 8701 raft_consensus.cc:2454] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9a4e138ca2b452c8b1f82e5c6cf36fc in term 1.
I20260925 15:25:37.567062 8910 leader_election.cc:302] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9a4e138ca2b452c8b1f82e5c6cf36fc, bac9df4f3fcf43c4afe9c113d9b0d79b; no voters:
I20260925 15:25:37.567170 9127 raft_consensus.cc:2781] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:37.567345 9127 raft_consensus.cc:686] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 LEADER]: Becoming Leader. State: Replica: b9a4e138ca2b452c8b1f82e5c6cf36fc, State: Running, Role: LEADER
I20260925 15:25:37.567438 9127 consensus_queue.cc:237] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } }
I20260925 15:25:37.567804 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.567650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:60429)
0925 15:25:37.567738 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.567778 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:25:37.568217 8381 catalog_manager.cc:5495] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc reported cstate change: term changed from 0 to 1, leader changed from <none> to b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67). New cstate: current_term: 1 leader_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9a4e138ca2b452c8b1f82e5c6cf36fc" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 46707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:37.577929 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.575410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.1.163.67:46707)
0925 15:25:37.575460 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:37.575462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:37.575472 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:37.575521 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:37.575522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:37.575540 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:37.575620 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:37.575734 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.576209 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.577109 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:37.577302 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:37.577377 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:37.577559 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:37.577675 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:37.577793 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:37.577810 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 15:25:37.577832 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:25:37.579164 8838 raft_consensus.cc:1261] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Refusing update from remote peer b9a4e138ca2b452c8b1f82e5c6cf36fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:37.579164 8701 raft_consensus.cc:1261] T 097d762c5f5e4da2b5e81c60e0ccf842 P b336feea90e546059515bc83bffffcec [term 1 FOLLOWER]: Refusing update from remote peer b9a4e138ca2b452c8b1f82e5c6cf36fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:37.579461 9127 consensus_queue.cc:1035] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac9df4f3fcf43c4afe9c113d9b0d79b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:37.579622 9129 consensus_queue.cc:1035] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b336feea90e546059515bc83bffffcec" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:37.580863 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.580740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:40019)
0925 15:25:37.580780 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.580815 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:25:37.581346 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.581235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:53695)
0925 15:25:37.581280 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.581308 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:25:37.582176 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.582065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:35367)
0925 15:25:37.582102 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.582142 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 15:25:37.582197 9135 mvcc.cc:205] Tried to move back new op lower bound from 7333273344321912832 to 7333273344276959232. Current Snapshot: MvccSnapshot[applied={T|T < 7333273344321912832}]
I20260925 15:25:37.582839 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.582743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.65:37953)
0925 15:25:37.582779 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.582806 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:25:37.582978 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.582890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.66:51557)
0925 15:25:37.582931 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.582952 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:25:37.583212 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.583112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36199 (local address 127.1.163.67:49329)
0925 15:25:37.583151 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.583183 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 15:25:37.585268 8382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:37.585381 8382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:37.586683 8382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3513a97b69314325830b8b2c75b0e6dd: Sending DeleteTablet for 3 replicas of tablet 097d762c5f5e4da2b5e81c60e0ccf842
I20260925 15:25:37.587066 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:37 UTC) from {username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:37800
I20260925 15:25:37.587088 8818 tablet_service.cc:1514] Processing DeleteTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:37 UTC) from {username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:57416
I20260925 15:25:37.587397 9143 tablet_replica.cc:330] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: stopping tablet replica
I20260925 15:25:37.587428 8681 tablet_service.cc:1514] Processing DeleteTablet for tablet 097d762c5f5e4da2b5e81c60e0ccf842 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:37 UTC) from {username='oryx', principal='oryx/127.1.163.125@KRBTEST.COM'} at 127.0.0.1:50582
I20260925 15:25:37.587455 9143 raft_consensus.cc:2229] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:37.587502 9143 raft_consensus.cc:2258] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:37.587828 9143 ts_tablet_manager.cc:1905] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:37.587991 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8529
I20260925 15:25:37.590045 9143 ts_tablet_manager.cc:1918] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:37.590145 9143 log.cc:1196] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/097d762c5f5e4da2b5e81c60e0ccf842
I20260925 15:25:37.590391 9143 ts_tablet_manager.cc:1939] T 097d762c5f5e4da2b5e81c60e0ccf842 P bac9df4f3fcf43c4afe9c113d9b0d79b: Deleting consensus metadata
I20260925 15:25:37.590749 8369 catalog_manager.cc:4841] TS bac9df4f3fcf43c4afe9c113d9b0d79b (127.1.163.66:39311): tablet 097d762c5f5e4da2b5e81c60e0ccf842 (table test-table [id=89172a76c912400e9b8024d863c475cf]) successfully deleted
I20260925 15:25:37.592316 9142 tablet_replica.cc:330] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: stopping tablet replica
I20260925 15:25:37.592404 9142 raft_consensus.cc:2229] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:37.592509 9142 raft_consensus.cc:2258] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:37.592875 9142 ts_tablet_manager.cc:1905] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:37.594589 9142 ts_tablet_manager.cc:1918] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:37.594657 9142 log.cc:1196] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/097d762c5f5e4da2b5e81c60e0ccf842
I20260925 15:25:37.594938 9142 ts_tablet_manager.cc:1939] T 097d762c5f5e4da2b5e81c60e0ccf842 P b9a4e138ca2b452c8b1f82e5c6cf36fc: Deleting consensus metadata
I20260925 15:25:37.595317 8366 catalog_manager.cc:4841] TS b9a4e138ca2b452c8b1f82e5c6cf36fc (127.1.163.67:46707): tablet 097d762c5f5e4da2b5e81c60e0ccf842 (table test-table [id=89172a76c912400e9b8024d863c475cf]) successfully deleted
I20260925 15:25:37.598495 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:37.598373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35985 (local address 127.0.0.1:50588)
0925 15:25:37.598437 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:37.598462 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:25:37.598671 8369 catalog_manager.cc:4584] TS b336feea90e546059515bc83bffffcec (127.1.163.65:35985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 097d762c5f5e4da2b5e81c60e0ccf842: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35985: connect: Connection refused (error 111)
I20260925 15:25:37.598694 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8755
I20260925 15:25:37.604821 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8891
I20260925 15:25:37.610312 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8248
I20260925 15:25:37.615197 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 8436
2026-09-25T15:25:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6675 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:37 dist-test-slave-hvv5 krb5kdc[9147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:37 dist-test-slave-hvv5 krb5kdc[9147](info): set up 2 sockets
Sep 25 15:25:37 dist-test-slave-hvv5 krb5kdc[9147](info): commencing operation
krb5kdc: starting...
W20260925 15:25:38.681319 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:38 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab.
Entry for principal oryx/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab.
I20260925 15:25:38.808938 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43365
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab
--principal=oryx/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:43365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:38.891218 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:38.891474 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:38.891511 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:38.892774 9160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:38.892835 9160 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:25:38.892861 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:38.892941 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:38.892971 9160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:38.893000 9160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:38.894541 9160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:43365
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.126.keytab
--principal=oryx/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:38.894825 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:38.895061 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:38.900023 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:38.900004 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:38.900033 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:38.900420 9160 server_base.cc:1034] running on GCE node
I20260925 15:25:38.900715 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:38.900985 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:38.902158 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790349938902136 us; error 32 us; skew 500 ppm
Sep 25 15:25:38 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349938, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:38.905056 9160 init.cc:377] Logged in from keytab as oryx/127.1.163.126@KRBTEST.COM (short username oryx)
I20260925 15:25:38.905583 9160 webserver.cc:460] Webserver started at http://127.1.163.126:37865/ using document root <none> and password file <none>
I20260925 15:25:38.905823 9160 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:38.905875 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:38.906001 9160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:38.907127 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "7315a8cfeed64119910abb7e662f578e"
format_stamp: "Formatted at 2026-09-25 15:25:38 on dist-test-slave-hvv5"
server_key: "99c8a0c9eaf6eac1914968a730b96fdd"
server_key_iv: "2fabe2bac02ee0aa8cd92aed48a246fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:38.907500 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "7315a8cfeed64119910abb7e662f578e"
format_stamp: "Formatted at 2026-09-25 15:25:38 on dist-test-slave-hvv5"
server_key: "99c8a0c9eaf6eac1914968a730b96fdd"
server_key_iv: "2fabe2bac02ee0aa8cd92aed48a246fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:38.908974 9160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:38.909953 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:38.910254 9160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:38.910408 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "7315a8cfeed64119910abb7e662f578e"
format_stamp: "Formatted at 2026-09-25 15:25:38 on dist-test-slave-hvv5"
server_key: "99c8a0c9eaf6eac1914968a730b96fdd"
server_key_iv: "2fabe2bac02ee0aa8cd92aed48a246fb"
server_key_version: "encryptionkey@0"
I20260925 15:25:38.910494 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:38.930249 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:38.932690 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:38.932895 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:38.938208 9160 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:43365
I20260925 15:25:38.938301 9237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:43365 every 8 connection(s)
I20260925 15:25:38.938668 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:38.939388 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:38.941998 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Bootstrap starting.
I20260925 15:25:38.942657 9238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:38.942996 9238 log.cc:824] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:38.943820 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: No bootstrap required, opened a new log
I20260925 15:25:38.944577 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9160
I20260925 15:25:38.944687 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:38.944871 1677 external_mini_cluster.cc:1428] Setting key b3e28ae3c0dcc0ebbb63428d1a9345f7
I20260925 15:25:38.946859 9238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } }
I20260925 15:25:38.947080 9238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:38.947203 9238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7315a8cfeed64119910abb7e662f578e, State: Initialized, Role: FOLLOWER
I20260925 15:25:38.947366 9238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } }
I20260925 15:25:38.947470 9238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:38.947507 9238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:38.947563 9238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:38.948393 9238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } }
I20260925 15:25:38.948489 9238 leader_election.cc:302] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7315a8cfeed64119910abb7e662f578e; no voters:
I20260925 15:25:38.948697 9238 leader_election.cc:288] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:38.948760 9243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:38.948913 9243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [term 1 LEADER]: Becoming Leader. State: Replica: 7315a8cfeed64119910abb7e662f578e, State: Running, Role: LEADER
I20260925 15:25:38.949000 9238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:38.949000 9243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } }
I20260925 15:25:38.949365 9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7315a8cfeed64119910abb7e662f578e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } } }
I20260925 15:25:38.949453 9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:38.949707 9245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7315a8cfeed64119910abb7e662f578e. Latest consensus state: current_term: 1 leader_uuid: "7315a8cfeed64119910abb7e662f578e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7315a8cfeed64119910abb7e662f578e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43365 } } }
I20260925 15:25:38.950044 9245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e [sys.catalog]: This master's current role is: LEADER
W20260925 15:25:38.950958 9259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:25:38.951015 9259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:25:38.951493 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:38.952099 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:25:38 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:38.954327 9249 catalog_manager.cc:1347] Generated new cluster ID: f9e48c6eb76044c78c356a6a04b7c5c8
I20260925 15:25:38.954386 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:38.959321 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:38.945431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.1.163.126:43365)
0925 15:25:38.945647 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:38.945655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:38.945681 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:25:38.946869 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:38.946878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:38.946967 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:38.947177 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:38.948058 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:38.949390 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:38.950594 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:38.951190 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:38.951311 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:38.954306 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:38.954310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:38.954311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:38.954323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:38.957409 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:38.957913 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:38.957917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:38.957917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:38.958032 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:38.958291 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:38.958292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:38.958292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:38.958619 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:38.958785 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:38.958921 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:25:38.959069 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260925 15:25:38.988288 9249 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:38.988935 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:38.997931 9249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Generated new TSK 0
I20260925 15:25:38.998162 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
I20260925 15:25:39.040856 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43365
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:39.136451 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:39.136698 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:39.136734 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:39.137884 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:39.137920 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:39.137941 9264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:39.137959 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:39.139439 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:43365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:39.139648 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:39.139843 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:39.145048 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:39.145057 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:39.145169 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:39.145323 9264 server_base.cc:1034] running on GCE node
I20260925 15:25:39.145597 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:39.145860 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:39.147035 9264 hybrid_clock.cc:648] HybridClock initialized: now 1790349939146985 us; error 59 us; skew 500 ppm
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:39.150112 9264 init.cc:377] Logged in from keytab as oryx/127.1.163.65@KRBTEST.COM (short username oryx)
I20260925 15:25:39.150599 9264 webserver.cc:460] Webserver started at http://127.1.163.65:35135/ using document root <none> and password file <none>
I20260925 15:25:39.150884 9264 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:39.150933 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:39.151081 9264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:39.152264 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "53c38e318161429f9288b0be9498fc2d"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "e7c5ef4c60f79eef5856eae4803f3930"
server_key_iv: "87d1bcf97a134becff322eeac678ef88"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.152665 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "53c38e318161429f9288b0be9498fc2d"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "e7c5ef4c60f79eef5856eae4803f3930"
server_key_iv: "87d1bcf97a134becff322eeac678ef88"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.154039 9264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:39.154826 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.155036 9264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:39.155122 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "53c38e318161429f9288b0be9498fc2d"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "e7c5ef4c60f79eef5856eae4803f3930"
server_key_iv: "87d1bcf97a134becff322eeac678ef88"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.155176 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:39.162569 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:39.165001 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:39.165126 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:39.165596 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:39.165635 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.165663 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:39.165721 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.172214 9264 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:40389
I20260925 15:25:39.172375 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:40389 every 8 connection(s)
I20260925 15:25:39.172712 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:39.176450 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.166277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:46917 (local address 127.1.163.126:43365)
0925 15:25:39.166339 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.166342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.166573 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:25:39.167260 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.167261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.167277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.167354 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.168228 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.169116 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.170695 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.171078 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.171156 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.173913 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.173915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.173915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.173925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.175239 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.175625 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.175626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.175626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.175675 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.175927 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.175927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.175927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.176032 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.176084 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.176317 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.176386 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:25:39.176636 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.166231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.65:46917)
0925 15:25:39.166423 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.166470 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.167118 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.167442 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.167446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.167831 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.168116 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.168122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.169222 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.169223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.170596 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.170598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.171210 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.171211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.171375 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.171992 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.172007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.173817 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.175329 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.175335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.175336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.175558 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.175734 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.175735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.175736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.175837 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.176089 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.176093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.176207 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.176291 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.176413 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260925 15:25:39.176905 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9264
I20260925 15:25:39.177021 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:39.177238 1677 external_mini_cluster.cc:1428] Setting key cdefc5664addb4c5727cc0ceaa15131a
I20260925 15:25:39.177366 9395 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43365
I20260925 15:25:39.177466 9395 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:39.177665 9395 heartbeater.cc:511] Master 127.1.163.126:43365 requested a full tablet report, sending...
I20260925 15:25:39.178329 9192 ts_manager.cc:194] Registered new tserver with Master: 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389)
I20260925 15:25:39.179196 9192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:46917
I20260925 15:25:39.185248 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.178463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:60799 (local address 127.1.163.126:43365)
0925 15:25:39.178503 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.178505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.178760 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:25:39.178833 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.178834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.178844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.178893 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.179203 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.179943 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.181087 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.181278 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.181340 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.182438 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.182440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.182440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.182447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.184065 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.184434 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.184435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.184435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.184493 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.184715 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.184716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.184717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.184834 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.184912 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.185067 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.185151 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:25:39.185268 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.178536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.65:60799)
0925 15:25:39.178681 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.178692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.178774 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.178896 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.178897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.179075 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.179124 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.179126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.180050 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.180051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.180971 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.180974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.181368 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.181369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.181443 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.182000 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.182005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.182326 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.184151 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.184153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.184153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.184344 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.184577 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.184578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.184579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.184647 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.184904 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.184905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.184954 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.185042 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.185151 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for oryx/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
I20260925 15:25:39.204066 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43365
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:39.290863 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:39.291121 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:39.291152 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:39.292670 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:39.292768 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:39.292794 9404 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:39.292815 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:39.294549 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:43365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:39.294945 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:39.295200 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:39.300607 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:39.300695 9404 server_base.cc:1034] running on GCE node
W20260925 15:25:39.300637 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:39.300637 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:39.301139 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:39.301375 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:39.302523 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790349939302491 us; error 49 us; skew 500 ppm
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:39.305555 9404 init.cc:377] Logged in from keytab as oryx/127.1.163.66@KRBTEST.COM (short username oryx)
I20260925 15:25:39.306121 9404 webserver.cc:460] Webserver started at http://127.1.163.66:38207/ using document root <none> and password file <none>
I20260925 15:25:39.306380 9404 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:39.306443 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:39.306633 9404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:39.307791 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "15afb2965897433bad9339d8dd2a9839"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "942cd71ab6c904da68fd9f57184243b6"
server_key_iv: "86c77fda8b1120b2e96d23a49f285fb9"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.308202 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "15afb2965897433bad9339d8dd2a9839"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "942cd71ab6c904da68fd9f57184243b6"
server_key_iv: "86c77fda8b1120b2e96d23a49f285fb9"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.309828 9404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:39.310762 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.311014 9404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:39.311102 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "15afb2965897433bad9339d8dd2a9839"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "942cd71ab6c904da68fd9f57184243b6"
server_key_iv: "86c77fda8b1120b2e96d23a49f285fb9"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.311160 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:39.320416 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:39.322696 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:39.322898 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:39.323482 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:39.323553 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.323623 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:39.323657 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.331471 9404 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:35487
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:39.331794 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:35487 every 8 connection(s)
I20260925 15:25:39.331933 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:39.335472 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.324064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:50161 (local address 127.1.163.126:43365)
0925 15:25:39.324109 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.324111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.324366 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:25:39.325038 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.325040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.325054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.325116 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.326131 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.326967 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.328766 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.329091 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.329179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.332192 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.332194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.332195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.332204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.333819 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.334367 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.334369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.334369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.334431 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.334757 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.334759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.334759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.334923 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.335001 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.335324 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.335405 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:25:39.335651 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.324063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.66:50161)
0925 15:25:39.324227 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.324262 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.324902 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.325239 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.325245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.325667 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.326018 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.326026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.327086 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.327087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.328616 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.328619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.329204 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.329206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.329377 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.330115 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.330126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.332070 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.333924 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.333932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.333934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.334193 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.334541 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.334542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.334543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.334656 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.335023 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.335030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.335166 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.335287 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.335432 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260925 15:25:39.336606 9535 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43365
I20260925 15:25:39.336717 9535 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:39.337073 9535 heartbeater.cc:511] Master 127.1.163.126:43365 requested a full tablet report, sending...
I20260925 15:25:39.337679 9191 ts_manager.cc:194] Registered new tserver with Master: 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487)
I20260925 15:25:39.338096 9191 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:50161
I20260925 15:25:39.339589 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9404
I20260925 15:25:39.339809 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:39.340036 1677 external_mini_cluster.cc:1428] Setting key be06fd309ce32ef042d7b57d3268699c
I20260925 15:25:39.345274 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.338391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.66:53005)
0925 15:25:39.338540 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.338551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.338630 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.338766 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.338766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.338970 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.339025 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.339027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.339703 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.339704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.340412 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.340414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.340826 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.340827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.340892 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.341510 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.341517 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.341858 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.344046 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.344047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.344047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.344231 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.344468 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.344469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.344469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.344529 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.344971 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.344972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.345021 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.345091 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.345195 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260925 15:25:39.345391 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.338430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:53005 (local address 127.1.163.126:43365)
0925 15:25:39.338478 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.338480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.338614 (+ 134us) server_negotiation.cc:407] Connection header received
0925 15:25:39.338692 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.338693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.338703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.338759 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.339116 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.339603 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.340489 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.340734 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.340826 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.342034 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.342035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.342036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.342043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.343935 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.344317 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.344319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.344319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.344395 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.344645 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.344646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.344647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.344869 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.344944 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.345143 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.345246 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for oryx/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
I20260925 15:25:39.372632 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:43365
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:39.455729 9544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:39.455996 9544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:39.456040 9544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:39.457413 9544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:39.457477 9544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:39.457516 9544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:25:39.457572 9544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:39.459255 9544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.163.126:43365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:39.459532 9544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:39.459777 9544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:39.465238 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:39.465230 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:39.465277 9544 server_base.cc:1034] running on GCE node
W20260925 15:25:39.465215 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:39.465718 9544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:39.466038 9544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:39.467211 9544 hybrid_clock.cc:648] HybridClock initialized: now 1790349939467186 us; error 41 us; skew 500 ppm
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:39.469925 9544 init.cc:377] Logged in from keytab as oryx/127.1.163.67@KRBTEST.COM (short username oryx)
I20260925 15:25:39.470378 9544 webserver.cc:460] Webserver started at http://127.1.163.67:45189/ using document root <none> and password file <none>
I20260925 15:25:39.470623 9544 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:39.470706 9544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:39.470852 9544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:39.471923 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "db72cdee9d6e499fbe994dd6ebe945da"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "5464bb83429331b8aa2ae291a2a9d2d0"
server_key_iv: "05ce52912ce34f1c535d132ed1e3a1e6"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.472293 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "db72cdee9d6e499fbe994dd6ebe945da"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "5464bb83429331b8aa2ae291a2a9d2d0"
server_key_iv: "05ce52912ce34f1c535d132ed1e3a1e6"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.473767 9544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:25:39.475067 9559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.475350 9544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:39.475462 9544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "db72cdee9d6e499fbe994dd6ebe945da"
format_stamp: "Formatted at 2026-09-25 15:25:39 on dist-test-slave-hvv5"
server_key: "5464bb83429331b8aa2ae291a2a9d2d0"
server_key_iv: "05ce52912ce34f1c535d132ed1e3a1e6"
server_key_version: "encryptionkey@0"
I20260925 15:25:39.475557 9544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:39.484921 9544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:39.487288 9544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:39.487460 9544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:39.487991 9544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:39.488058 9544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.488103 9544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:39.488145 9544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.494956 9544 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:38253
I20260925 15:25:39.495087 9674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:38253 every 8 connection(s)
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:39.495362 9544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:25:39.498003 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9544
I20260925 15:25:39.498116 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:39.498327 1677 external_mini_cluster.cc:1428] Setting key 7e4e91a968b91b928000c8bb8883f8fa
I20260925 15:25:39.499094 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.488598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:41131 (local address 127.1.163.126:43365)
0925 15:25:39.488680 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.488683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.488894 (+ 211us) server_negotiation.cc:407] Connection header received
0925 15:25:39.489388 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.489389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.489403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.489481 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.490438 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.491311 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.492929 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.493257 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.493351 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.495938 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.495939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.495940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.495947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.497510 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.498036 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.498037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.498038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.498113 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.498392 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.498393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.498394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.498554 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.498620 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.498916 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.499005 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:25:39.499275 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.488521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.67:41131)
0925 15:25:39.488704 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.488745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.489283 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.489577 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.489581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.490028 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.490319 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.490326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.491414 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.491416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.492830 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.492833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.493373 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.493374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.493534 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.494163 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.494175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.495833 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.497624 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.497630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.497633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.497915 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.498193 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.498194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.498194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.498318 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.498611 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.498617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.498770 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.498880 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.499045 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260925 15:25:39.499768 9675 heartbeater.cc:347] Connected to a master server at 127.1.163.126:43365
I20260925 15:25:39.499855 9675 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:39.500051 9675 heartbeater.cc:511] Master 127.1.163.126:43365 requested a full tablet report, sending...
I20260925 15:25:39.500429 9192 ts_manager.cc:194] Registered new tserver with Master: db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:39.501004 9192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:41131
I20260925 15:25:39.506320 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.500916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40733 (local address 127.1.163.126:43365)
0925 15:25:39.500958 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.500960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.501025 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:25:39.501110 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.501111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.501121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.501169 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.501499 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.502078 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.502790 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.502965 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.503018 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.503926 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.503927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.503927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.503932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.505253 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.505582 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.505583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.505583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.505630 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.505835 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.505836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.505836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.505930 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.505982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.506190 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.506264 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:25:39.506325 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.500834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.1.163.67:40733)
0925 15:25:39.500967 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.500978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.501040 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.501194 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.501195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.501362 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.501425 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.501426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.502167 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.502168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.502676 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.502677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.503032 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.503033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.503081 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.503559 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.503564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.503852 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.505330 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.505331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.505331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.505498 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.505714 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.505715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.505715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.505774 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.506020 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.506021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.506062 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.506151 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.506248 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260925 15:25:39.510141 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:25:39.532413 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:39.526284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.1.163.126:43365)
0925 15:25:39.526358 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.526360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.526369 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:39.526394 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.526395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.526405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.526477 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.527129 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.527643 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.528485 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.528806 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.528887 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.532296 (+ 3409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38260: BlockingRecv error: recv got EOF from 127.0.0.1:38260 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:39.541390 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.533563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.1.163.126:43365)
0925 15:25:39.533627 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.533629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.533641 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:25:39.533724 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.533725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.533734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.533805 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.534313 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.534814 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.535614 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.535832 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.535941 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.538640 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.538642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.538642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.538650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.540149 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.540605 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.540607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.540608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.540671 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.540894 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.540895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.540895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.541018 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.541101 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.541233 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.541306 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:25:39.543591 9192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:39.545388 9192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:39.552615 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.548033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.1.163.66:35487)
0925 15:25:39.548314 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.548318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.548445 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:25:39.548527 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.548529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.548575 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.548650 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:39.548850 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.549786 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.551359 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.552114 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.552201 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.552284 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.552423 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.552462 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
I20260925 15:25:39.552624 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.547881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:35487 (local address 127.0.0.1:50644)
0925 15:25:39.548359 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.548367 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.548457 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.548715 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.548716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.548718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:39.548769 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.548771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.549882 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.549883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.551267 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.551270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.552197 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.552198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.552296 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.552312 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.552402 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.552467 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":391,"thread_start_us":66,"threads_started":1}
I20260925 15:25:39.552624 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.547881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38253 (local address 127.0.0.1:42464)
0925 15:25:39.548266 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.548277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.548322 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.548633 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.548633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.548635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:39.548676 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.548677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.550049 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.550050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.551382 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.551384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.552240 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.552241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.552304 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.552315 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.552391 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.552467 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":72,"thread_start_us":98,"threads_started":1}
I20260925 15:25:39.553072 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.548123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.1.163.67:38253)
0925 15:25:39.548448 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.548452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.548462 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:39.548498 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.548500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.548513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.548745 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:39.548797 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.549945 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.551468 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.552168 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.552232 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.552298 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.552745 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.552964 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":87,"threads_started":1}
I20260925 15:25:39.553407 9469 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 (DEFAULT_TABLE table=test-table [id=f1f8d347cd574d7ea0e94525486de143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.553790 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6bd84a4e040868af50536c14ecfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.553887 9609 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 (DEFAULT_TABLE table=test-table [id=f1f8d347cd574d7ea0e94525486de143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.554203 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6bd84a4e040868af50536c14ecfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.556981 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.547881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:40389 (local address 127.0.0.1:45862)
0925 15:25:39.548053 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.548094 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.548159 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.548504 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.548507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.548510 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:39.548640 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.548642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.550028 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.550029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.551395 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.551396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.556634 (+ 5238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.556636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.556708 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.556712 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.556845 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.556897 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":166,"thread_start_us":62,"threads_started":1}
I20260925 15:25:39.557014 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.547989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.1.163.65:40389)
0925 15:25:39.548322 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.548328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.548338 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:39.548378 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.548380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.548396 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.548725 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:39.548787 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.549945 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.552335 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.553439 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.553511 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.553607 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.556867 (+ 3260us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.556912 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":75,"threads_started":1}
I20260925 15:25:39.557797 9703 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Bootstrap starting.
I20260925 15:25:39.557960 9329 tablet_service.cc:1467] Processing CreateTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 (DEFAULT_TABLE table=test-table [id=f1f8d347cd574d7ea0e94525486de143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.558230 9704 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Bootstrap starting.
I20260925 15:25:39.558337 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbc6bd84a4e040868af50536c14ecfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.558748 9703 tablet_bootstrap.cc:654] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.558890 9704 tablet_bootstrap.cc:654] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.559134 9703 log.cc:824] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:39.559211 9704 log.cc:824] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:39.560000 9703 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: No bootstrap required, opened a new log
I20260925 15:25:39.560077 9703 ts_tablet_manager.cc:1397] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:39.560866 9704 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: No bootstrap required, opened a new log
I20260925 15:25:39.560953 9704 ts_tablet_manager.cc:1397] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:25:39.562275 9703 raft_consensus.cc:348] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.562448 9703 raft_consensus.cc:374] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.562493 9703 raft_consensus.cc:729] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.562407 9704 raft_consensus.cc:348] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.562573 9704 raft_consensus.cc:374] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.562623 9704 raft_consensus.cc:729] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.562608 9703 consensus_queue.cc:260] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.562546 9707 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Bootstrap starting.
I20260925 15:25:39.562744 9704 consensus_queue.cc:260] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.562877 9703 ts_tablet_manager.cc:1428] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:25:39.562885 9675 heartbeater.cc:503] Master 127.1.163.126:43365 was elected leader, sending a full tablet report...
I20260925 15:25:39.563000 9704 ts_tablet_manager.cc:1428] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:39.563140 9535 heartbeater.cc:503] Master 127.1.163.126:43365 was elected leader, sending a full tablet report...
I20260925 15:25:39.563483 9707 tablet_bootstrap.cc:654] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.563825 9707 log.cc:824] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:39.564651 9707 tablet_bootstrap.cc:492] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: No bootstrap required, opened a new log
I20260925 15:25:39.564724 9707 ts_tablet_manager.cc:1397] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:39.566066 9707 raft_consensus.cc:348] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.566212 9707 raft_consensus.cc:374] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.566257 9707 raft_consensus.cc:729] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c38e318161429f9288b0be9498fc2d, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.566377 9707 consensus_queue.cc:260] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.566630 9707 ts_tablet_manager.cc:1428] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:25:39.566682 9395 heartbeater.cc:503] Master 127.1.163.126:43365 was elected leader, sending a full tablet report...
W20260925 15:25:39.582623 9536 tablet.cc:2369] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:39.670758 9709 raft_consensus.cc:482] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:39.670921 9709 raft_consensus.cc:504] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.671284 9709 leader_election.cc:288] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
W20260925 15:25:39.673487 9396 tablet.cc:2369] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:39.675359 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.671597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:51827 (local address 127.1.163.67:38253)
0925 15:25:39.671651 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.671653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.671757 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:25:39.671840 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.671842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.671852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.671906 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:39.672066 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.672776 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.674352 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.674925 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.675017 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.675028 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.675243 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.675290 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:25:39.675359 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.671551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:39693 (local address 127.1.163.65:40389)
0925 15:25:39.671597 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.671600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.671610 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:39.671689 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.671690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.671706 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.671771 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:39.671967 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.672764 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.674351 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.674962 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.675027 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.675035 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.675238 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.675286 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:25:39.675369 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.671466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:40389 (local address 127.1.163.66:39693)
0925 15:25:39.671526 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.671545 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.671601 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.671789 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.671790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.671791 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:39.671871 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.671873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.672866 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.672867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.674219 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.674223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.675039 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.675040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.675133 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.675135 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.675198 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.675253 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":90}
I20260925 15:25:39.675380 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.671502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38253 (local address 127.1.163.66:51827)
0925 15:25:39.671672 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.671684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.671775 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.671930 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.671931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.671933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:39.672005 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.672007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.672881 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.672882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.674219 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.674223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.675039 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.675040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.675131 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.675135 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.675198 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.675253 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":64,"thread_start_us":125,"threads_started":1}
I20260925 15:25:39.675761 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6bd84a4e040868af50536c14ecfb0" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d" is_pre_election: true
I20260925 15:25:39.675899 9349 raft_consensus.cc:2454] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:39.676156 9422 leader_election.cc:302] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, 53c38e318161429f9288b0be9498fc2d; no voters:
I20260925 15:25:39.676175 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6bd84a4e040868af50536c14ecfb0" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da" is_pre_election: true
I20260925 15:25:39.676326 9629 raft_consensus.cc:2454] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:39.676326 9709 raft_consensus.cc:2781] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:39.676427 9709 raft_consensus.cc:482] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:39.676462 9709 raft_consensus.cc:3037] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.677587 9709 raft_consensus.cc:504] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.677771 9709 leader_election.cc:288] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Requested vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:39.677968 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6bd84a4e040868af50536c14ecfb0" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da"
I20260925 15:25:39.678053 9629 raft_consensus.cc:3037] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.678020 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbc6bd84a4e040868af50536c14ecfb0" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d"
I20260925 15:25:39.678119 9349 raft_consensus.cc:3037] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.678836 9629 raft_consensus.cc:2454] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:39.678946 9349 raft_consensus.cc:2454] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:39.679009 9420 leader_election.cc:302] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:39.679116 9709 raft_consensus.cc:2781] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:39.679281 9709 raft_consensus.cc:686] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Becoming Leader. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Running, Role: LEADER
I20260925 15:25:39.679399 9709 consensus_queue.cc:237] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.680286 9191 catalog_manager.cc:5495] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15afb2965897433bad9339d8dd2a9839 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "15afb2965897433bad9339d8dd2a9839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:39.698091 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.695478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.1.163.66:35487)
0925 15:25:39.695541 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.695543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.695559 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:25:39.695595 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.695595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.695613 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.695674 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:39.695809 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.696289 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.697195 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.697439 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.697496 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.697722 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:39.697836 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:39.697936 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.697950 (+ 14us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.697969 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:25:39.700026 9629 raft_consensus.cc:1261] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:39.700001 9349 raft_consensus.cc:1261] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:39.700390 9713 consensus_queue.cc:1035] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:39.700556 9709 consensus_queue.cc:1035] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:39.703065 9716 mvcc.cc:205] Tried to move back new op lower bound from 7333273353008078848 to 7333273352927424512. Current Snapshot: MvccSnapshot[applied={T|T < 7333273353008078848}]
I20260925 15:25:39.705833 9191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:39.705955 9191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:39.706883 9191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Sending DeleteTablet for 3 replicas of tablet fbc6bd84a4e040868af50536c14ecfb0
I20260925 15:25:39.707201 9329 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:39 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:45862
I20260925 15:25:39.707329 9469 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:39 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50644
I20260925 15:25:39.707453 9191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:25:39.707543 9609 tablet_service.cc:1514] Processing DeleteTablet for tablet fbc6bd84a4e040868af50536c14ecfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:39 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42464
W20260925 15:25:39.707603 9191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:39.707782 9727 tablet_replica.cc:330] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: stopping tablet replica
I20260925 15:25:39.707901 9727 raft_consensus.cc:2229] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:39.708015 9727 raft_consensus.cc:2258] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:39.708181 9726 tablet_replica.cc:330] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: stopping tablet replica
I20260925 15:25:39.708277 9725 tablet_replica.cc:330] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: stopping tablet replica
I20260925 15:25:39.708446 9725 raft_consensus.cc:2229] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:39.708510 9727 ts_tablet_manager.cc:1905] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:39.709116 9725 raft_consensus.cc:2258] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:39.709345 9609 tablet_service.cc:1467] Processing CreateTablet for tablet b7745ca94323420dbfbd07751fe7bac8 (DEFAULT_TABLE table=test-table [id=3d8083802476428084089a797991bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.709465 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7745ca94323420dbfbd07751fe7bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.710482 9703 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Bootstrap starting.
I20260925 15:25:39.708665 9726 raft_consensus.cc:2229] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:39.709267 9469 tablet_service.cc:1467] Processing CreateTablet for tablet b7745ca94323420dbfbd07751fe7bac8 (DEFAULT_TABLE table=test-table [id=3d8083802476428084089a797991bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.710935 9703 tablet_bootstrap.cc:654] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.710964 9726 raft_consensus.cc:2258] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:39.710973 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7745ca94323420dbfbd07751fe7bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.709283 9329 tablet_service.cc:1467] Processing CreateTablet for tablet b7745ca94323420dbfbd07751fe7bac8 (DEFAULT_TABLE table=test-table [id=3d8083802476428084089a797991bfd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:39.711169 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7745ca94323420dbfbd07751fe7bac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.712160 9707 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Bootstrap starting.
I20260925 15:25:39.712571 9707 tablet_bootstrap.cc:654] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.712793 9727 ts_tablet_manager.cc:1918] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:39.712862 9727 log.cc:1196] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/fbc6bd84a4e040868af50536c14ecfb0
I20260925 15:25:39.712924 9704 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Bootstrap starting.
I20260925 15:25:39.713120 9727 ts_tablet_manager.cc:1939] T fbc6bd84a4e040868af50536c14ecfb0 P db72cdee9d6e499fbe994dd6ebe945da: Deleting consensus metadata
I20260925 15:25:39.713323 9704 tablet_bootstrap.cc:654] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.713461 9726 ts_tablet_manager.cc:1905] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:39.713601 9725 ts_tablet_manager.cc:1905] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:39.713604 9176 catalog_manager.cc:4841] TS db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253): tablet fbc6bd84a4e040868af50536c14ecfb0 (table test-table [id=f1f8d347cd574d7ea0e94525486de143]) successfully deleted
I20260925 15:25:39.714057 9703 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: No bootstrap required, opened a new log
I20260925 15:25:39.714108 9703 ts_tablet_manager.cc:1397] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:25:39.714145 9704 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: No bootstrap required, opened a new log
I20260925 15:25:39.714185 9704 ts_tablet_manager.cc:1397] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:39.714277 9703 raft_consensus.cc:348] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.714352 9703 raft_consensus.cc:374] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.714327 9704 raft_consensus.cc:348] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.714372 9703 raft_consensus.cc:729] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.714392 9704 raft_consensus.cc:374] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.714412 9704 raft_consensus.cc:729] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.714414 9703 consensus_queue.cc:260] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.714453 9704 consensus_queue.cc:260] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.714533 9704 ts_tablet_manager.cc:1428] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.714530 9703 ts_tablet_manager.cc:1428] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:39.716329 9726 ts_tablet_manager.cc:1918] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:39.716408 9726 log.cc:1196] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/fbc6bd84a4e040868af50536c14ecfb0
I20260925 15:25:39.716706 9726 ts_tablet_manager.cc:1939] T fbc6bd84a4e040868af50536c14ecfb0 P 15afb2965897433bad9339d8dd2a9839: Deleting consensus metadata
I20260925 15:25:39.716719 9725 ts_tablet_manager.cc:1918] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:39.716801 9725 log.cc:1196] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/fbc6bd84a4e040868af50536c14ecfb0
I20260925 15:25:39.717057 9725 ts_tablet_manager.cc:1939] T fbc6bd84a4e040868af50536c14ecfb0 P 53c38e318161429f9288b0be9498fc2d: Deleting consensus metadata
I20260925 15:25:39.717128 9179 catalog_manager.cc:4841] TS 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487): tablet fbc6bd84a4e040868af50536c14ecfb0 (table test-table [id=f1f8d347cd574d7ea0e94525486de143]) successfully deleted
I20260925 15:25:39.717499 9178 catalog_manager.cc:4841] TS 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389): tablet fbc6bd84a4e040868af50536c14ecfb0 (table test-table [id=f1f8d347cd574d7ea0e94525486de143]) successfully deleted
I20260925 15:25:39.717737 9707 tablet_bootstrap.cc:492] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: No bootstrap required, opened a new log
I20260925 15:25:39.717793 9707 ts_tablet_manager.cc:1397] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 15:25:39.717927 9707 raft_consensus.cc:348] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.718008 9707 raft_consensus.cc:374] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.718029 9707 raft_consensus.cc:729] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c38e318161429f9288b0be9498fc2d, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.718075 9707 consensus_queue.cc:260] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.718185 9707 ts_tablet_manager.cc:1428] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260925 15:25:39.746032 9676 tablet.cc:2369] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:25:39.873803 9709 raft_consensus.cc:482] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:39.873991 9709 raft_consensus.cc:504] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.874154 9709 leader_election.cc:288] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:39.874390 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7745ca94323420dbfbd07751fe7bac8" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da" is_pre_election: true
I20260925 15:25:39.874390 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7745ca94323420dbfbd07751fe7bac8" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d" is_pre_election: true
I20260925 15:25:39.874504 9629 raft_consensus.cc:2454] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:39.874504 9349 raft_consensus.cc:2454] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:39.874679 9422 leader_election.cc:302] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, 53c38e318161429f9288b0be9498fc2d; no voters:
I20260925 15:25:39.874848 9709 raft_consensus.cc:2781] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:39.874912 9709 raft_consensus.cc:482] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:39.874938 9709 raft_consensus.cc:3037] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.875677 9709 raft_consensus.cc:504] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.875835 9709 leader_election.cc:288] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Requested vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:39.875990 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7745ca94323420dbfbd07751fe7bac8" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da"
I20260925 15:25:39.876116 9629 raft_consensus.cc:3037] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.876413 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7745ca94323420dbfbd07751fe7bac8" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d"
I20260925 15:25:39.876503 9349 raft_consensus.cc:3037] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:39.877013 9629 raft_consensus.cc:2454] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:39.877247 9420 leader_election.cc:302] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:39.877365 9709 raft_consensus.cc:2781] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:39.877380 9349 raft_consensus.cc:2454] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:39.877420 9709 raft_consensus.cc:686] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Becoming Leader. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Running, Role: LEADER
I20260925 15:25:39.877487 9709 consensus_queue.cc:237] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.878165 9192 catalog_manager.cc:5495] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15afb2965897433bad9339d8dd2a9839 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "15afb2965897433bad9339d8dd2a9839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:25:39 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349938, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.126@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:39.924842 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.917389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:43365 (local address 127.0.0.1:38276)
0925 15:25:39.917537 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:39.917556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:39.917618 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:39.917796 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:39.917797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:39.918140 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:39.918191 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.918193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.918878 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.918879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.919415 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:39.919417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.919772 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.919773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:39.919817 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.920335 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:39.920340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:39.921868 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:39.923405 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.923413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.923415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.923623 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.923923 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:39.923923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:39.923923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:39.923985 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:39.924389 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:39.924396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:39.924444 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:25:39.924514 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:25:39.924745 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260925 15:25:39.925024 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:39.917491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.1.163.126:43365)
0925 15:25:39.917583 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:39.917584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:39.917600 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:25:39.917689 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:39.917690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:39.917703 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:39.917771 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:39.918291 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.918774 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.919512 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:39.919693 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:39.919757 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:39.921970 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:39.921971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:39.921972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:39.921979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:39.923319 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.923719 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.923720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.923721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.923839 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:39.924075 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:39.924076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:39.924076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:39.924198 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:39.924362 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:39.924621 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:25:39.924902 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":60}
I20260925 15:25:39.927390 9192 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:38276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:25:39.927718 9192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:39.929512 9469 tablet_service.cc:1467] Processing CreateTablet for tablet 416d7b2dda354bdbb3be3dd80e59683c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2ff59af37ac40dda436c124dfec8280]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:39.929554 9329 tablet_service.cc:1467] Processing CreateTablet for tablet 416d7b2dda354bdbb3be3dd80e59683c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2ff59af37ac40dda436c124dfec8280]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:39.929626 9609 tablet_service.cc:1467] Processing CreateTablet for tablet 416d7b2dda354bdbb3be3dd80e59683c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2ff59af37ac40dda436c124dfec8280]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:25:39.929697 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416d7b2dda354bdbb3be3dd80e59683c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.929754 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416d7b2dda354bdbb3be3dd80e59683c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.929756 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416d7b2dda354bdbb3be3dd80e59683c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:39.930958 9707 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: Bootstrap starting.
I20260925 15:25:39.931434 9707 tablet_bootstrap.cc:654] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.931581 9704 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: Bootstrap starting.
I20260925 15:25:39.931917 9704 tablet_bootstrap.cc:654] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.932214 9707 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: No bootstrap required, opened a new log
I20260925 15:25:39.932240 9703 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: Bootstrap starting.
I20260925 15:25:39.932292 9707 ts_tablet_manager.cc:1397] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:39.932467 9707 raft_consensus.cc:348] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.932585 9707 raft_consensus.cc:374] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.932616 9707 raft_consensus.cc:729] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c38e318161429f9288b0be9498fc2d, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.932721 9703 tablet_bootstrap.cc:654] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:39.932683 9707 consensus_queue.cc:260] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.932842 9707 ts_tablet_manager.cc:1428] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:39.932934 9711 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:39.933069 9711 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:39.933228 9704 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: No bootstrap required, opened a new log
I20260925 15:25:39.933303 9704 ts_tablet_manager.cc:1397] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:39.933435 9704 raft_consensus.cc:348] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.933516 9704 raft_consensus.cc:374] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.933547 9704 raft_consensus.cc:729] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.933601 9704 consensus_queue.cc:260] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.933702 9704 ts_tablet_manager.cc:1428] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.933738 9709 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:39.933868 9709 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:39.934194 9703 tablet_bootstrap.cc:492] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: No bootstrap required, opened a new log
I20260925 15:25:39.934291 9703 ts_tablet_manager.cc:1397] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:25:39.934465 9703 raft_consensus.cc:348] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.934542 9703 raft_consensus.cc:374] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:39.934556 9703 raft_consensus.cc:729] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Initialized, Role: FOLLOWER
I20260925 15:25:39.934621 9703 consensus_queue.cc:260] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:39.934716 9703 ts_tablet_manager.cc:1428] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:39.934774 9708 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:39.934855 9708 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:40.126542 9708 raft_consensus.cc:482] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:40.126670 9708 raft_consensus.cc:504] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.127058 9708 leader_election.cc:288] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487)
I20260925 15:25:40.130326 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.127359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:37591 (local address 127.1.163.65:40389)
0925 15:25:40.127410 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.127412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.127550 (+ 138us) server_negotiation.cc:407] Connection header received
0925 15:25:40.127630 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.127631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.127660 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.127716 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:40.127941 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.128412 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.129603 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.129945 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.130001 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.130008 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.130207 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.130252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:25:40.130370 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.127295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:35487 (local address 127.1.163.67:52027)
0925 15:25:40.127425 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.127436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.127507 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.127686 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.127687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.127689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:40.127770 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.127774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.128641 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.128641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.129450 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.129453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.130102 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.130103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.130154 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.130157 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.130231 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.130272 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260925 15:25:40.130370 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.127362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:52027 (local address 127.1.163.66:35487)
0925 15:25:40.127422 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.127424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.127540 (+ 116us) server_negotiation.cc:407] Connection header received
0925 15:25:40.127581 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.127582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.127600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.127676 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:25:40.127871 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.128442 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.129553 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.129998 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.130085 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.130095 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.130255 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.130295 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:25:40.130399 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.127296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:40389 (local address 127.1.163.67:37591)
0925 15:25:40.127433 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.127444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.127550 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.127743 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.127744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.127746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:25:40.127848 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.127850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.128502 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.128503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.129463 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.129466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.130031 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.130032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.130116 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.130122 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.130192 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.130274 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":53,"thread_start_us":62,"threads_started":1}
I20260925 15:25:40.130615 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416d7b2dda354bdbb3be3dd80e59683c" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb2965897433bad9339d8dd2a9839" is_pre_election: true
I20260925 15:25:40.130724 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416d7b2dda354bdbb3be3dd80e59683c" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d" is_pre_election: true
I20260925 15:25:40.130816 9489 raft_consensus.cc:2454] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 0.
I20260925 15:25:40.130842 9349 raft_consensus.cc:2454] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 0.
I20260925 15:25:40.131054 9563 leader_election.cc:302] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.131208 9708 raft_consensus.cc:2781] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:40.131251 9708 raft_consensus.cc:482] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:40.131280 9708 raft_consensus.cc:3037] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.131971 9708 raft_consensus.cc:504] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.132120 9708 leader_election.cc:288] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 election: Requested vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487)
I20260925 15:25:40.132280 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416d7b2dda354bdbb3be3dd80e59683c" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb2965897433bad9339d8dd2a9839"
I20260925 15:25:40.132345 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416d7b2dda354bdbb3be3dd80e59683c" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d"
I20260925 15:25:40.132400 9489 raft_consensus.cc:3037] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.132411 9349 raft_consensus.cc:3037] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.133183 9349 raft_consensus.cc:2454] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 1.
I20260925 15:25:40.133303 9489 raft_consensus.cc:2454] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 1.
I20260925 15:25:40.133442 9562 leader_election.cc:302] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c38e318161429f9288b0be9498fc2d, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.133620 9708 raft_consensus.cc:2781] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:40.133765 9708 raft_consensus.cc:686] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [term 1 LEADER]: Becoming Leader. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Running, Role: LEADER
I20260925 15:25:40.133841 9708 consensus_queue.cc:237] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.134229 9743 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: TxnStatusTablet state changed. Reason: New leader db72cdee9d6e499fbe994dd6ebe945da. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.134341 9743 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:40.134516 9745 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:25:40.134624 9745 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:25:40.134624 9192 catalog_manager.cc:5495] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da reported cstate change: term changed from 0 to 1, leader changed from <none> to db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67). New cstate: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } health_report { overall_health: HEALTHY } } }
I20260925 15:25:40.168398 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.164708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38253 (local address 127.0.0.1:42478)
0925 15:25:40.164853 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.164865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.164917 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.165261 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.165262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.165263 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:40.165319 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.165322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.166296 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.166297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.167346 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.167349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.167808 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.167809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.167883 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.167909 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.168211 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.168222 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.168278 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.168331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 15:25:40.168388 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.165075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.1.163.67:38253)
0925 15:25:40.165116 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.165118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.165125 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:40.165164 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.165165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.165186 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.165250 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.165383 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.166211 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.167450 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.167717 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.167806 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.168021 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.168139 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.168225 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.168284 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.168305 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:25:40.169613 9349 raft_consensus.cc:1261] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Refusing update from remote peer db72cdee9d6e499fbe994dd6ebe945da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.169893 9489 raft_consensus.cc:1261] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Refusing update from remote peer db72cdee9d6e499fbe994dd6ebe945da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.169885 9708 consensus_queue.cc:1035] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.170490 9711 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: TxnStatusTablet state changed. Reason: New leader db72cdee9d6e499fbe994dd6ebe945da. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.170588 9711 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:40.170565 9708 consensus_queue.cc:1035] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Connected to new peer: Peer: permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.171131 9709 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: TxnStatusTablet state changed. Reason: New leader db72cdee9d6e499fbe994dd6ebe945da. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.171222 9709 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:40.171511 9708 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.171590 9708 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:40.171674 9708 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.171718 9708 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P db72cdee9d6e499fbe994dd6ebe945da: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:25:40.171779 9717 mvcc.cc:205] Tried to move back new op lower bound from 7333273354932555776 to 7333273354788868096. Current Snapshot: MvccSnapshot[applied={T|T < 7333273354932555776}]
I20260925 15:25:40.172016 9709 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.172092 9709 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 15afb2965897433bad9339d8dd2a9839: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:40.173329 9718 mvcc.cc:205] Tried to move back new op lower bound from 7333273354932555776 to 7333273354788868096. Current Snapshot: MvccSnapshot[applied={T|T < 7333273354932555776}]
I20260925 15:25:40.173532 9711 tablet_replica.cc:439] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } } }
I20260925 15:25:40.173614 9711 tablet_replica.cc:442] T 416d7b2dda354bdbb3be3dd80e59683c P 53c38e318161429f9288b0be9498fc2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:25:40.179677 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.176485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:50043 (local address 127.1.163.67:38253)
0925 15:25:40.176532 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.176553 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.176602 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:25:40.176670 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.176670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.176681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.176748 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.176837 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.177635 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.178750 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.179052 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.179125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.179369 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.179405 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.179459 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.179563 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.179583 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:25:40.179657 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.176419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38253 (local address 127.1.163.66:50043)
0925 15:25:40.176527 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.176559 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.176624 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.176750 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.176751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.176752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:40.176800 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.176801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.177713 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.177714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.178662 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.178665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.179158 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.179159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.179225 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.179243 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.179494 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.179497 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.179550 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.179596 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":54,"threads_started":1}
I20260925 15:25:40.181780 9349 raft_consensus.cc:1261] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.181850 9629 raft_consensus.cc:1261] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.182070 9713 consensus_queue.cc:1035] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.182186 9709 consensus_queue.cc:1035] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.190973 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.188024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:35487 (local address 127.1.163.67:51727)
0925 15:25:40.188159 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.188171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.188221 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.188398 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.188399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.188414 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:40.188493 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.188496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.189438 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.189439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.190159 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.190162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.190616 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.190617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.190665 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.190678 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.190845 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.190846 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.190876 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.190908 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260925 15:25:40.190973 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.188172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:51727 (local address 127.1.163.66:35487)
0925 15:25:40.188224 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.188226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.188234 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:40.188314 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.188315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.188326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.188388 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.188610 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.189358 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.190234 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.190539 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.190607 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.190753 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.190788 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.190846 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.190879 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.190900 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:25:40.199839 9192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.199972 9192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.200697 9192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Sending DeleteTablet for 3 replicas of tablet b7745ca94323420dbfbd07751fe7bac8
I20260925 15:25:40.201040 9609 tablet_service.cc:1514] Processing DeleteTablet for tablet b7745ca94323420dbfbd07751fe7bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42464
I20260925 15:25:40.201049 9329 tablet_service.cc:1514] Processing DeleteTablet for tablet b7745ca94323420dbfbd07751fe7bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:45862
I20260925 15:25:40.201138 9725 tablet_replica.cc:330] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: stopping tablet replica
I20260925 15:25:40.201143 9727 tablet_replica.cc:330] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: stopping tablet replica
I20260925 15:25:40.201191 9725 raft_consensus.cc:2229] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.201192 9727 raft_consensus.cc:2229] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.201227 9727 raft_consensus.cc:2258] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.201227 9725 raft_consensus.cc:2258] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.201537 9727 ts_tablet_manager.cc:1905] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.202210 9469 tablet_service.cc:1514] Processing DeleteTablet for tablet b7745ca94323420dbfbd07751fe7bac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50644
I20260925 15:25:40.202433 9725 ts_tablet_manager.cc:1905] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.203244 9726 tablet_replica.cc:330] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: stopping tablet replica
I20260925 15:25:40.203323 9726 raft_consensus.cc:2229] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:40.203392 9726 raft_consensus.cc:2258] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.203594 9727 ts_tablet_manager.cc:1918] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:40.203647 9727 log.cc:1196] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/b7745ca94323420dbfbd07751fe7bac8
I20260925 15:25:40.203728 9726 ts_tablet_manager.cc:1905] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.203893 9727 ts_tablet_manager.cc:1939] T b7745ca94323420dbfbd07751fe7bac8 P db72cdee9d6e499fbe994dd6ebe945da: Deleting consensus metadata
I20260925 15:25:40.204134 9725 ts_tablet_manager.cc:1918] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:40.204182 9725 log.cc:1196] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/b7745ca94323420dbfbd07751fe7bac8
I20260925 15:25:40.204236 9176 catalog_manager.cc:4841] TS db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253): tablet b7745ca94323420dbfbd07751fe7bac8 (table test-table [id=3d8083802476428084089a797991bfd7]) successfully deleted
I20260925 15:25:40.204396 9725 ts_tablet_manager.cc:1939] T b7745ca94323420dbfbd07751fe7bac8 P 53c38e318161429f9288b0be9498fc2d: Deleting consensus metadata
I20260925 15:25:40.204876 9178 catalog_manager.cc:4841] TS 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389): tablet b7745ca94323420dbfbd07751fe7bac8 (table test-table [id=3d8083802476428084089a797991bfd7]) successfully deleted
I20260925 15:25:40.205226 9726 ts_tablet_manager.cc:1918] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:40.205312 9726 log.cc:1196] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/b7745ca94323420dbfbd07751fe7bac8
I20260925 15:25:40.205521 9726 ts_tablet_manager.cc:1939] T b7745ca94323420dbfbd07751fe7bac8 P 15afb2965897433bad9339d8dd2a9839: Deleting consensus metadata
I20260925 15:25:40.206050 9179 catalog_manager.cc:4841] TS 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487): tablet b7745ca94323420dbfbd07751fe7bac8 (table test-table [id=3d8083802476428084089a797991bfd7]) successfully deleted
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:25:40.229671 9242 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:40.223372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.1.163.126:43365)
0925 15:25:40.223445 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.223448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.223552 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:25:40.223619 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.223621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.223634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.223696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:40.224259 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.225016 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.225815 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.226097 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.226202 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.229595 (+ 3393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38278: BlockingRecv error: recv got EOF from 127.0.0.1:38278 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM
I20260925 15:25:40.238351 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.230574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.1.163.126:43365)
0925 15:25:40.230629 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.230631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.230680 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:25:40.230809 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.230810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.230820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.230889 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:40.231393 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.231894 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.232669 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.232868 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.232948 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.235255 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:40.235257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:40.235257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:40.235272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:40.237077 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:40.237528 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:40.237529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:40.237530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:40.237586 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:40.237826 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:40.237826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:40.237827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:40.237944 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:40.238063 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.238201 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.238266 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:25:40.240253 9192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:40.240458 9192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:40.242012 9609 tablet_service.cc:1467] Processing CreateTablet for tablet f4651a106a4c40a3917ce13b41754b2f (DEFAULT_TABLE table=test-table [id=c68499ae469545faa9d7c24cf32df608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.242024 9469 tablet_service.cc:1467] Processing CreateTablet for tablet f4651a106a4c40a3917ce13b41754b2f (DEFAULT_TABLE table=test-table [id=c68499ae469545faa9d7c24cf32df608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.242074 9329 tablet_service.cc:1467] Processing CreateTablet for tablet f4651a106a4c40a3917ce13b41754b2f (DEFAULT_TABLE table=test-table [id=c68499ae469545faa9d7c24cf32df608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.242161 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4651a106a4c40a3917ce13b41754b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.242177 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4651a106a4c40a3917ce13b41754b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.242189 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4651a106a4c40a3917ce13b41754b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.243498 9703 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Bootstrap starting.
I20260925 15:25:40.243716 9704 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Bootstrap starting.
I20260925 15:25:40.243954 9703 tablet_bootstrap.cc:654] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.244064 9704 tablet_bootstrap.cc:654] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.244361 9707 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Bootstrap starting.
I20260925 15:25:40.244844 9707 tablet_bootstrap.cc:654] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.245110 9704 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: No bootstrap required, opened a new log
I20260925 15:25:40.245189 9704 ts_tablet_manager.cc:1397] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:40.245343 9704 raft_consensus.cc:348] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.245437 9704 raft_consensus.cc:374] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.245471 9704 raft_consensus.cc:729] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.245522 9704 consensus_queue.cc:260] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.245564 9707 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: No bootstrap required, opened a new log
I20260925 15:25:40.245615 9707 ts_tablet_manager.cc:1397] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:40.245631 9704 ts_tablet_manager.cc:1428] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:40.245769 9707 raft_consensus.cc:348] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.245852 9707 raft_consensus.cc:374] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.245877 9707 raft_consensus.cc:729] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c38e318161429f9288b0be9498fc2d, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.245924 9707 consensus_queue.cc:260] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.246012 9707 ts_tablet_manager.cc:1428] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:40.246436 9703 tablet_bootstrap.cc:492] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: No bootstrap required, opened a new log
I20260925 15:25:40.246493 9703 ts_tablet_manager.cc:1397] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:25:40.246656 9703 raft_consensus.cc:348] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.246743 9703 raft_consensus.cc:374] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.246774 9703 raft_consensus.cc:729] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.246819 9703 consensus_queue.cc:260] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.246907 9703 ts_tablet_manager.cc:1428] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:40.538230 9709 raft_consensus.cc:482] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:40.538358 9709 raft_consensus.cc:504] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.538566 9709 leader_election.cc:288] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:40.538920 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4651a106a4c40a3917ce13b41754b2f" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d" is_pre_election: true
I20260925 15:25:40.538920 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4651a106a4c40a3917ce13b41754b2f" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da" is_pre_election: true
I20260925 15:25:40.539027 9349 raft_consensus.cc:2454] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:40.539034 9629 raft_consensus.cc:2454] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 0.
I20260925 15:25:40.539224 9420 leader_election.cc:302] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.539355 9709 raft_consensus.cc:2781] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:40.539392 9709 raft_consensus.cc:482] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:40.539417 9709 raft_consensus.cc:3037] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.540125 9709 raft_consensus.cc:504] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.540280 9709 leader_election.cc:288] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Requested vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253)
I20260925 15:25:40.540457 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4651a106a4c40a3917ce13b41754b2f" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d"
I20260925 15:25:40.540457 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4651a106a4c40a3917ce13b41754b2f" candidate_uuid: "15afb2965897433bad9339d8dd2a9839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db72cdee9d6e499fbe994dd6ebe945da"
I20260925 15:25:40.540653 9349 raft_consensus.cc:3037] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.540659 9629 raft_consensus.cc:3037] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.541463 9349 raft_consensus.cc:2454] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:40.541464 9629 raft_consensus.cc:2454] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15afb2965897433bad9339d8dd2a9839 in term 1.
I20260925 15:25:40.541664 9420 leader_election.cc:302] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb2965897433bad9339d8dd2a9839, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.541839 9709 raft_consensus.cc:2781] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:40.541899 9709 raft_consensus.cc:686] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Becoming Leader. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Running, Role: LEADER
I20260925 15:25:40.541981 9709 consensus_queue.cc:237] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.542623 9191 catalog_manager.cc:5495] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15afb2965897433bad9339d8dd2a9839 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "15afb2965897433bad9339d8dd2a9839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } health_report { overall_health: UNKNOWN } } }
I20260925 15:25:40.602244 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.600126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.1.163.66:35487)
0925 15:25:40.600196 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.600198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.600206 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:40.600231 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.600232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.600246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.600336 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.600448 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.600943 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.601660 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.601853 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.601904 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.602037 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.602064 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.602102 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.602162 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.602182 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:25:40.603230 9349 raft_consensus.cc:1261] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.603283 9629 raft_consensus.cc:1261] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Refusing update from remote peer 15afb2965897433bad9339d8dd2a9839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.603616 9709 consensus_queue.cc:1035] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.603775 9755 consensus_queue.cc:1035] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.606261 9191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.606359 9191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.607069 9191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Sending DeleteTablet for 3 replicas of tablet f4651a106a4c40a3917ce13b41754b2f
I20260925 15:25:40.607331 9469 tablet_service.cc:1514] Processing DeleteTablet for tablet f4651a106a4c40a3917ce13b41754b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50644
I20260925 15:25:40.607357 9609 tablet_service.cc:1514] Processing DeleteTablet for tablet f4651a106a4c40a3917ce13b41754b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42464
I20260925 15:25:40.607437 9727 tablet_replica.cc:330] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: stopping tablet replica
I20260925 15:25:40.607438 9726 tablet_replica.cc:330] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: stopping tablet replica
I20260925 15:25:40.607484 9727 raft_consensus.cc:2229] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.607515 9726 raft_consensus.cc:2229] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:40.607529 9727 raft_consensus.cc:2258] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.607504 9329 tablet_service.cc:1514] Processing DeleteTablet for tablet f4651a106a4c40a3917ce13b41754b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:45862
I20260925 15:25:40.607571 9726 raft_consensus.cc:2258] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.607618 9725 tablet_replica.cc:330] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: stopping tablet replica
I20260925 15:25:40.607724 9725 raft_consensus.cc:2229] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.607771 9725 raft_consensus.cc:2258] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.608071 9725 ts_tablet_manager.cc:1905] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.608065 9727 ts_tablet_manager.cc:1905] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.608825 9191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:25:40.609158 9191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:25:40.609719 9725 ts_tablet_manager.cc:1918] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:40.609730 9726 ts_tablet_manager.cc:1905] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.609767 9725 log.cc:1196] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/f4651a106a4c40a3917ce13b41754b2f
I20260925 15:25:40.609942 9725 ts_tablet_manager.cc:1939] T f4651a106a4c40a3917ce13b41754b2f P 53c38e318161429f9288b0be9498fc2d: Deleting consensus metadata
I20260925 15:25:40.610226 9178 catalog_manager.cc:4841] TS 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389): tablet f4651a106a4c40a3917ce13b41754b2f (table test-table [id=c68499ae469545faa9d7c24cf32df608]) successfully deleted
I20260925 15:25:40.610612 9329 tablet_service.cc:1467] Processing CreateTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 (DEFAULT_TABLE table=test-table [id=469563f4587c494c90d3a3a846228504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.610708 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5772f5c0447299a9fcb7e7a29ee07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.610682 9469 tablet_service.cc:1467] Processing CreateTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 (DEFAULT_TABLE table=test-table [id=469563f4587c494c90d3a3a846228504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.610787 9469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5772f5c0447299a9fcb7e7a29ee07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.611233 9726 ts_tablet_manager.cc:1918] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:40.611295 9726 log.cc:1196] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/f4651a106a4c40a3917ce13b41754b2f
I20260925 15:25:40.611512 9726 ts_tablet_manager.cc:1939] T f4651a106a4c40a3917ce13b41754b2f P 15afb2965897433bad9339d8dd2a9839: Deleting consensus metadata
I20260925 15:25:40.611747 9727 ts_tablet_manager.cc:1918] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:25:40.611802 9609 tablet_service.cc:1467] Processing CreateTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 (DEFAULT_TABLE table=test-table [id=469563f4587c494c90d3a3a846228504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:25:40.611958 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5772f5c0447299a9fcb7e7a29ee07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:40.611946 9179 catalog_manager.cc:4841] TS 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487): tablet f4651a106a4c40a3917ce13b41754b2f (table test-table [id=c68499ae469545faa9d7c24cf32df608]) successfully deleted
I20260925 15:25:40.611990 9704 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Bootstrap starting.
I20260925 15:25:40.612003 9707 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: Bootstrap starting.
I20260925 15:25:40.611802 9727 log.cc:1196] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/f4651a106a4c40a3917ce13b41754b2f
I20260925 15:25:40.612273 9704 tablet_bootstrap.cc:654] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.612365 9727 ts_tablet_manager.cc:1939] T f4651a106a4c40a3917ce13b41754b2f P db72cdee9d6e499fbe994dd6ebe945da: Deleting consensus metadata
I20260925 15:25:40.612394 9707 tablet_bootstrap.cc:654] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.612779 9176 catalog_manager.cc:4841] TS db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253): tablet f4651a106a4c40a3917ce13b41754b2f (table test-table [id=c68499ae469545faa9d7c24cf32df608]) successfully deleted
I20260925 15:25:40.613006 9704 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: No bootstrap required, opened a new log
I20260925 15:25:40.613059 9704 ts_tablet_manager.cc:1397] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:40.613173 9704 raft_consensus.cc:348] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.613258 9704 raft_consensus.cc:374] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.613286 9704 raft_consensus.cc:729] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb2965897433bad9339d8dd2a9839, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.613337 9704 consensus_queue.cc:260] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.613387 9703 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Bootstrap starting.
I20260925 15:25:40.613445 9704 ts_tablet_manager.cc:1428] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:40.613687 9703 tablet_bootstrap.cc:654] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:40.613695 9707 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: No bootstrap required, opened a new log
I20260925 15:25:40.613737 9707 ts_tablet_manager.cc:1397] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:25:40.613905 9707 raft_consensus.cc:348] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.613967 9707 raft_consensus.cc:374] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.613986 9707 raft_consensus.cc:729] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c38e318161429f9288b0be9498fc2d, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.614024 9707 consensus_queue.cc:260] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.614110 9707 ts_tablet_manager.cc:1428] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:40.614535 9703 tablet_bootstrap.cc:492] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: No bootstrap required, opened a new log
I20260925 15:25:40.614590 9703 ts_tablet_manager.cc:1397] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:40.614727 9703 raft_consensus.cc:348] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.614796 9703 raft_consensus.cc:374] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:40.614810 9703 raft_consensus.cc:729] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Initialized, Role: FOLLOWER
I20260925 15:25:40.614866 9703 consensus_queue.cc:260] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.614979 9703 ts_tablet_manager.cc:1428] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:40.745496 9759 raft_consensus.cc:482] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:25:40.745616 9759 raft_consensus.cc:504] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.745822 9759 leader_election.cc:288] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487)
I20260925 15:25:40.746120 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5772f5c0447299a9fcb7e7a29ee07" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d" is_pre_election: true
I20260925 15:25:40.746133 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5772f5c0447299a9fcb7e7a29ee07" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb2965897433bad9339d8dd2a9839" is_pre_election: true
I20260925 15:25:40.746232 9349 raft_consensus.cc:2454] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 0.
I20260925 15:25:40.746232 9489 raft_consensus.cc:2454] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 0.
I20260925 15:25:40.746395 9562 leader_election.cc:302] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c38e318161429f9288b0be9498fc2d, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.746500 9759 raft_consensus.cc:2781] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:25:40.746546 9759 raft_consensus.cc:482] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:25:40.746572 9759 raft_consensus.cc:3037] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.747267 9759 raft_consensus.cc:504] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.747429 9759 leader_election.cc:288] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 election: Requested vote from peers 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389), 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487)
I20260925 15:25:40.747584 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5772f5c0447299a9fcb7e7a29ee07" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c38e318161429f9288b0be9498fc2d"
I20260925 15:25:40.747677 9349 raft_consensus.cc:3037] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.747649 9489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5772f5c0447299a9fcb7e7a29ee07" candidate_uuid: "db72cdee9d6e499fbe994dd6ebe945da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb2965897433bad9339d8dd2a9839"
I20260925 15:25:40.747751 9489 raft_consensus.cc:3037] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:40.748293 9349 raft_consensus.cc:2454] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 1.
I20260925 15:25:40.748410 9489 raft_consensus.cc:2454] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db72cdee9d6e499fbe994dd6ebe945da in term 1.
I20260925 15:25:40.748497 9562 leader_election.cc:302] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c38e318161429f9288b0be9498fc2d, db72cdee9d6e499fbe994dd6ebe945da; no voters:
I20260925 15:25:40.748643 9759 raft_consensus.cc:2781] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:40.748783 9759 raft_consensus.cc:686] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 1 LEADER]: Becoming Leader. State: Replica: db72cdee9d6e499fbe994dd6ebe945da, State: Running, Role: LEADER
I20260925 15:25:40.748847 9759 consensus_queue.cc:237] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } }
I20260925 15:25:40.749583 9192 catalog_manager.cc:5495] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da reported cstate change: term changed from 0 to 1, leader changed from <none> to db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67). New cstate: current_term: 1 leader_uuid: "db72cdee9d6e499fbe994dd6ebe945da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db72cdee9d6e499fbe994dd6ebe945da" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 38253 } health_report { overall_health: HEALTHY } } }
I20260925 15:25:40.766762 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.764317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.1.163.67:38253)
0925 15:25:40.764470 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.764473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.764480 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:25:40.764502 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.764502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.764520 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.764640 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.764752 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.765258 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.765986 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.766216 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.766298 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.766468 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.766511 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.766568 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.766634 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.766657 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260925 15:25:40.771319 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.768996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:33887 (local address 127.1.163.67:38253)
0925 15:25:40.769059 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:40.769062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:40.769154 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:25:40.769218 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:40.769219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:40.769230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:40.769280 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:25:40.769415 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.769891 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.770638 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.770826 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.770905 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.771042 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.771078 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.771139 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:40.771192 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 15:25:40.771231 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":41}
I20260925 15:25:40.771322 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.768917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:38253 (local address 127.1.163.67:33887)
0925 15:25:40.769077 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.769088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:40.769167 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:40.769282 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:40.769282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:40.769284 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:25:40.769330 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.769331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.769978 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.769979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.770541 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:40.770543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:40.770904 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:40.770905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:40.770956 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:40.770960 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:25:40.771140 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:25:40.771142 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:25:40.771191 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:25:40.771245 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260925 15:25:40.773545 9489 raft_consensus.cc:1261] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Refusing update from remote peer db72cdee9d6e499fbe994dd6ebe945da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.773573 9349 raft_consensus.cc:1261] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Refusing update from remote peer db72cdee9d6e499fbe994dd6ebe945da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:25:40.773780 9759 consensus_queue.cc:1035] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Connected to new peer: Peer: permanent_uuid: "15afb2965897433bad9339d8dd2a9839" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.773905 9782 consensus_queue.cc:1035] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c38e318161429f9288b0be9498fc2d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:25:40.784934 9192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.785071 9192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:25:40.785801 9192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7315a8cfeed64119910abb7e662f578e: Sending DeleteTablet for 3 replicas of tablet aef5772f5c0447299a9fcb7e7a29ee07
I20260925 15:25:40.786055 9329 tablet_service.cc:1514] Processing DeleteTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:45862
I20260925 15:25:40.786172 9608 tablet_service.cc:1514] Processing DeleteTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42464
I20260925 15:25:40.786186 9725 tablet_replica.cc:330] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d: stopping tablet replica
I20260925 15:25:40.786245 9725 raft_consensus.cc:2229] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.786260 9727 tablet_replica.cc:330] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: stopping tablet replica
I20260925 15:25:40.786283 9725 raft_consensus.cc:2258] T aef5772f5c0447299a9fcb7e7a29ee07 P 53c38e318161429f9288b0be9498fc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.786316 9727 raft_consensus.cc:2229] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:25:40.786412 9727 raft_consensus.cc:2258] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.786640 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9264
I20260925 15:25:40.786630 9469 tablet_service.cc:1514] Processing DeleteTablet for tablet aef5772f5c0447299a9fcb7e7a29ee07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:25:40 UTC) from {username='oryx', principal='oryx/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50644
I20260925 15:25:40.786717 9726 tablet_replica.cc:330] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: stopping tablet replica
I20260925 15:25:40.786763 9726 raft_consensus.cc:2229] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:25:40.786840 9726 raft_consensus.cc:2258] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:25:40.789330 9726 ts_tablet_manager.cc:1905] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.790594 9727 ts_tablet_manager.cc:1905] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:25:40.791039 9726 ts_tablet_manager.cc:1918] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:40.791112 9726 log.cc:1196] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/aef5772f5c0447299a9fcb7e7a29ee07
I20260925 15:25:40.791347 9726 ts_tablet_manager.cc:1939] T aef5772f5c0447299a9fcb7e7a29ee07 P 15afb2965897433bad9339d8dd2a9839: Deleting consensus metadata
I20260925 15:25:40.791707 9179 catalog_manager.cc:4841] TS 15afb2965897433bad9339d8dd2a9839 (127.1.163.66:35487): tablet aef5772f5c0447299a9fcb7e7a29ee07 (table test-table [id=469563f4587c494c90d3a3a846228504]) successfully deleted
I20260925 15:25:40.792207 9727 ts_tablet_manager.cc:1918] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:25:40.792258 9727 log.cc:1196] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/aef5772f5c0447299a9fcb7e7a29ee07
I20260925 15:25:40.792500 9727 ts_tablet_manager.cc:1939] T aef5772f5c0447299a9fcb7e7a29ee07 P db72cdee9d6e499fbe994dd6ebe945da: Deleting consensus metadata
I20260925 15:25:40.792874 9176 catalog_manager.cc:4841] TS db72cdee9d6e499fbe994dd6ebe945da (127.1.163.67:38253): tablet aef5772f5c0447299a9fcb7e7a29ee07 (table test-table [id=469563f4587c494c90d3a3a846228504]) successfully deleted
W20260925 15:25:40.796275 9178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:40389 (error 108)
I20260925 15:25:40.796897 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:40.796670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:40389 (local address 127.0.0.1:45868)
0925 15:25:40.796800 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:40.796851 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260925 15:25:40.796936 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9404
W20260925 15:25:40.797019 9178 catalog_manager.cc:4584] TS 53c38e318161429f9288b0be9498fc2d (127.1.163.65:40389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aef5772f5c0447299a9fcb7e7a29ee07: Network error: Client connection negotiation failed: client connection to 127.1.163.65:40389: connect: Connection refused (error 111)
I20260925 15:25:40.805127 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9544
I20260925 15:25:40.812680 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9160
2026-09-25T15:25:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3196 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9797](info): set up 2 sockets
Sep 25 15:25:40 dist-test-slave-hvv5 krb5kdc[9797](info): commencing operation
krb5kdc: starting...
W20260925 15:25:41.865345 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:41 dist-test-slave-hvv5 krb5kdc[9797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:41.991482 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34999
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:42167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:34999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:42.074303 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:42.074586 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:42.074626 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:42.075879 9810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:42.075949 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:42.075976 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:42.076005 9810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:42.076036 9810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:42.077696 9810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:42167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:34999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:42.077996 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:42.078239 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:42.083214 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:42.083235 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:42.083274 9810 server_base.cc:1034] running on GCE node
W20260925 15:25:42.083214 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:42.083748 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:42.084064 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:42.085248 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790349942085238 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:25:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1276 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:25:42 dist-test-slave-hvv5 krb5kdc[9823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:25:42 dist-test-slave-hvv5 krb5kdc[9823](info): set up 2 sockets
Sep 25 15:25:42 dist-test-slave-hvv5 krb5kdc[9823](info): commencing operation
krb5kdc: starting...
W20260925 15:25:43.145344 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:25:43.275921 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36453
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:43.360052 9836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:43.360284 9836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:43.360309 9836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:43.361555 9836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:43.361605 9836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:43.361619 9836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:43.361632 9836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:43.361645 9836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:43.363268 9836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:43.363543 9836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:43.363780 9836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:43.368798 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.368824 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.368800 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:43.369143 9836 server_base.cc:1034] running on GCE node
I20260925 15:25:43.369410 9836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:43.369714 9836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:43.370880 9836 hybrid_clock.cc:648] HybridClock initialized: now 1790349943370864 us; error 30 us; skew 500 ppm
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:43.380834 9836 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:43.381299 9836 webserver.cc:460] Webserver started at http://127.1.163.126:41059/ using document root <none> and password file <none>
I20260925 15:25:43.381565 9836 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:43.381614 9836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:43.381801 9836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:43.382948 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "80bb8c712030458192d0dedbae7350e4"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "17279c705a00749a363baa7786b5310e"
server_key_iv: "481cc9aa9f5c7a51357f67c6e5f3178e"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.383327 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "80bb8c712030458192d0dedbae7350e4"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "17279c705a00749a363baa7786b5310e"
server_key_iv: "481cc9aa9f5c7a51357f67c6e5f3178e"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.384904 9836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:43.385723 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.385952 9836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:43.386059 9836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "80bb8c712030458192d0dedbae7350e4"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "17279c705a00749a363baa7786b5310e"
server_key_iv: "481cc9aa9f5c7a51357f67c6e5f3178e"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.386137 9836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:43.396713 9836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:43.399194 9836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:43.399369 9836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:43.403394 9836 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36453
I20260925 15:25:43.403473 9903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36453 every 8 connection(s)
I20260925 15:25:43.403829 9836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:43.404814 9904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:43.407130 9904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Bootstrap starting.
I20260925 15:25:43.407745 9904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:43.408085 9904 log.cc:824] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:43.408938 9904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: No bootstrap required, opened a new log
I20260925 15:25:43.410789 9904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:43.410938 9904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:43.410961 9904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bb8c712030458192d0dedbae7350e4, State: Initialized, Role: FOLLOWER
I20260925 15:25:43.411116 9904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:43.411218 9904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:43.411257 9904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:43.411322 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9836
I20260925 15:25:43.411339 9904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:43.411449 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:43.411633 1677 external_mini_cluster.cc:1428] Setting key 3d0db65a702a5eb01c11805dac9f1b24
I20260925 15:25:43.412034 9904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:43.412146 9904 leader_election.cc:302] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80bb8c712030458192d0dedbae7350e4; no voters:
I20260925 15:25:43.412317 9904 leader_election.cc:288] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:43.412585 9909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:43.412735 9904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:43.412803 9909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 LEADER]: Becoming Leader. State: Replica: 80bb8c712030458192d0dedbae7350e4, State: Running, Role: LEADER
I20260925 15:25:43.412962 9909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:43.413329 9909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bb8c712030458192d0dedbae7350e4. Latest consensus state: current_term: 1 leader_uuid: "80bb8c712030458192d0dedbae7350e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } } }
I20260925 15:25:43.413481 9909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:43.413990 9918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:43.414115 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80bb8c712030458192d0dedbae7350e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } } }
I20260925 15:25:43.414363 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:43.414644 9918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:43.416209 9918 catalog_manager.cc:1347] Generated new cluster ID: 0944c7a27a964552b227cc5baac4d538
I20260925 15:25:43.416263 9918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:43.424448 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.412165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.1.163.126:36453)
0925 15:25:43.412356 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:43.412366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:43.412404 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:25:43.412563 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:43.412573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:43.412636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:43.412796 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:43.413646 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.415128 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.416155 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.416704 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.416828 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.419541 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:43.419549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:43.419549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:43.419567 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:43.422659 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.423125 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.423128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.423129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.423257 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.423490 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.423491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.423491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.423766 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:43.423931 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:43.424048 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:25:43.424204 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260925 15:25:43.435322 9918 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:43.435880 9918 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:43.450421 9918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Generated new TSK 0
I20260925 15:25:43.450650 9918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:25:43.484480 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:43.592657 9930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:43.592886 9930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:43.592912 9930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:43.594098 9930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:43.594136 9930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:43.594163 9930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:43.595569 9930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:43.595772 9930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:43.595960 9930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:43.601354 9936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.601362 9935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.601367 9938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:43.601380 9930 server_base.cc:1034] running on GCE node
I20260925 15:25:43.601799 9930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:43.602072 9930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:43.603282 9930 hybrid_clock.cc:648] HybridClock initialized: now 1790349943603278 us; error 38 us; skew 500 ppm
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:43.606201 9930 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:25:43.606722 9930 webserver.cc:460] Webserver started at http://127.1.163.65:32907/ using document root <none> and password file <none>
I20260925 15:25:43.606976 9930 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:43.607023 9930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:43.607192 9930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:43.608299 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "0c63b0670cc84dbdb64ddfc15882469c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "cb3415e35bb84186925472987c304cb4"
server_key_iv: "be97f3f73b3ed1342e22942013e5218c"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.608690 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "0c63b0670cc84dbdb64ddfc15882469c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "cb3415e35bb84186925472987c304cb4"
server_key_iv: "be97f3f73b3ed1342e22942013e5218c"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.610123 9930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:43.611071 9945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.611263 9930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:43.611371 9930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "0c63b0670cc84dbdb64ddfc15882469c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "cb3415e35bb84186925472987c304cb4"
server_key_iv: "be97f3f73b3ed1342e22942013e5218c"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.611457 9930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:43.629424 9930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:43.631552 9930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:43.631729 9930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:43.631989 9930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:43.632318 9930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:43.632371 9930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.632413 9930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:43.632441 9930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.638480 9930 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:33421
I20260925 15:25:43.638526 10058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:33421 every 8 connection(s)
I20260925 15:25:43.638878 9930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:43.640750 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 9930
I20260925 15:25:43.640867 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:43.641052 1677 external_mini_cluster.cc:1428] Setting key e11e3fc971926bacb87e58b2561a669e
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:43.650041 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.639609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42133 (local address 127.1.163.126:36453)
0925 15:25:43.639680 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:43.639682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:43.639925 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:25:43.640432 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:43.640433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:43.640447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:43.640511 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:43.641850 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.642356 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.643787 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.644033 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.644103 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.646730 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:43.646733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:43.646733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:43.646743 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:43.648483 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.648960 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.648962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.648962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.649022 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.649292 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.649293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.649294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.649459 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:43.649543 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:43.649820 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:25:43.649925 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:25:43.650193 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.639589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:42133)
0925 15:25:43.639754 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:43.639799 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:43.640312 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:43.640659 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:43.640663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:43.641471 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:43.641742 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.641749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.642453 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.642454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.643683 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.643686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.644155 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.644156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.644330 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.644932 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:43.644942 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:43.646608 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:43.648622 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.648629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.648631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.648875 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.649095 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.649096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.649096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.649222 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.649537 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:43.649543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:43.649672 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:25:43.649764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:25:43.649906 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:25:43.650902 10059 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36453
I20260925 15:25:43.650997 10059 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:43.651196 10059 heartbeater.cc:511] Master 127.1.163.126:36453 requested a full tablet report, sending...
I20260925 15:25:43.651880 9868 ts_manager.cc:194] Registered new tserver with Master: 0c63b0670cc84dbdb64ddfc15882469c (127.1.163.65:33421)
I20260925 15:25:43.652472 9868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:42133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:25:43.671794 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:43.754837 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:43.755082 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:43.755108 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:43.756349 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:43.756403 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:43.756431 10064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:43.757965 10064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:43.758210 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:43.758473 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:43.763686 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:43.763835 10064 server_base.cc:1034] running on GCE node
W20260925 15:25:43.763703 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.763701 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:43.764175 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:43.764472 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:43.765641 10064 hybrid_clock.cc:648] HybridClock initialized: now 1790349943765639 us; error 56 us; skew 500 ppm
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:43.768247 10064 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:25:43.768776 10064 webserver.cc:460] Webserver started at http://127.1.163.66:36271/ using document root <none> and password file <none>
I20260925 15:25:43.769025 10064 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:43.769095 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:43.769251 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:43.770334 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "e3dfb0b258c9467da23f1bbc1bee190c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "ceb9f78cea66c2e888dbb0d18ba7f62d"
server_key_iv: "dacf4b73f768a5f772dcc24f9dc597bc"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.770694 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "e3dfb0b258c9467da23f1bbc1bee190c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "ceb9f78cea66c2e888dbb0d18ba7f62d"
server_key_iv: "dacf4b73f768a5f772dcc24f9dc597bc"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.772015 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:43.772850 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.773025 10064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:43.773124 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "e3dfb0b258c9467da23f1bbc1bee190c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "ceb9f78cea66c2e888dbb0d18ba7f62d"
server_key_iv: "dacf4b73f768a5f772dcc24f9dc597bc"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.773207 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:43.781981 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:43.783987 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:43.784158 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:43.784401 10064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:43.784822 10064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:43.784880 10064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.784924 10064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:43.784952 10064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.791205 10064 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:43421
I20260925 15:25:43.791257 10192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:43421 every 8 connection(s)
I20260925 15:25:43.791592 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:43.797312 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10064
I20260925 15:25:43.797457 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:25:43.797668 1677 external_mini_cluster.cc:1428] Setting key e493dda6c04ce8c2a2f19afba18ddc07
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:43.801976 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.792332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40907 (local address 127.1.163.126:36453)
0925 15:25:43.792399 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:43.792401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:43.792662 (+ 261us) server_negotiation.cc:407] Connection header received
0925 15:25:43.793184 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:43.793186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:43.793198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:43.793247 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:43.794198 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.794720 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.795733 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.795949 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.796029 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.798608 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:43.798610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:43.798610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:43.798618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:43.800286 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.800948 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.800950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.800950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.801007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.801315 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.801316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.801316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.801437 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:43.801487 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:43.801834 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 15:25:43.801909 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:25:43.802269 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.792301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:40907)
0925 15:25:43.792487 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:43.792530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:43.793075 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:43.793317 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:43.793321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:43.793897 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:43.794087 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.794093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.794819 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.794820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.795633 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.795635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.796036 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.796037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.796173 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.796803 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:43.796812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:43.798504 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:43.800389 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.800399 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.800401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.800851 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.801079 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.801080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.801080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.801233 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.801500 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:43.801506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:43.801684 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 15:25:43.801801 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:25:43.801985 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260925 15:25:43.802773 10193 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36453
I20260925 15:25:43.802888 10193 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:43.803102 10193 heartbeater.cc:511] Master 127.1.163.126:36453 requested a full tablet report, sending...
I20260925 15:25:43.803505 9868 ts_manager.cc:194] Registered new tserver with Master: e3dfb0b258c9467da23f1bbc1bee190c (127.1.163.66:43421)
I20260925 15:25:43.803913 9868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:40907
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:25:43.828740 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:43.914757 10198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:43.914963 10198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:43.914987 10198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:43.916168 10198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:43.916210 10198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:43.916239 10198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:43.918071 10198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:43.918322 10198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:43.918537 10198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:43.924086 10204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.924067 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:43.924084 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:43.924523 10198 server_base.cc:1034] running on GCE node
I20260925 15:25:43.924799 10198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:43.925072 10198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:43.926249 10198 hybrid_clock.cc:648] HybridClock initialized: now 1790349943926227 us; error 43 us; skew 500 ppm
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:43.929019 10198 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:25:43.929520 10198 webserver.cc:460] Webserver started at http://127.1.163.67:40595/ using document root <none> and password file <none>
I20260925 15:25:43.929772 10198 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:43.929849 10198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:43.929996 10198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:43.931095 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "04c768a7f93d4323a3e8bfd047d3d787"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "bbe54cf91c14089934aa55a88576f55a"
server_key_iv: "8091dd2ad64710ee3db47d9953b341a9"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.931460 10198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "04c768a7f93d4323a3e8bfd047d3d787"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "bbe54cf91c14089934aa55a88576f55a"
server_key_iv: "8091dd2ad64710ee3db47d9953b341a9"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.932852 10198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:25:43.933614 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.933812 10198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:43.933892 10198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "04c768a7f93d4323a3e8bfd047d3d787"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "bbe54cf91c14089934aa55a88576f55a"
server_key_iv: "8091dd2ad64710ee3db47d9953b341a9"
server_key_version: "encryptionkey@0"
I20260925 15:25:43.933949 10198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:43.965235 10198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:43.967726 10198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:43.967859 10198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:43.968058 10198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:43.968390 10198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:43.968448 10198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.968492 10198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:43.968516 10198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:43.974448 10198 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:36703
I20260925 15:25:43.974504 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:36703 every 8 connection(s)
I20260925 15:25:43.974887 10198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:43 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349943, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:25:43.984455 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.975634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:35731 (local address 127.1.163.126:36453)
0925 15:25:43.975697 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:43.975700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:43.976037 (+ 337us) server_negotiation.cc:407] Connection header received
0925 15:25:43.976517 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:43.976519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:43.976530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:43.976640 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:43.977786 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.978307 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.979322 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.979504 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.979557 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.981912 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:43.981913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:43.981913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:43.981920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:43.983189 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.983611 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.983612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.983612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.983674 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:43.983902 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:43.983902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:43.983903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:43.984014 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:43.984067 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:43.984299 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:25:43.984387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:25:43.984555 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10198
I20260925 15:25:43.984644 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:25:43.984726 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:43.975617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:35731)
0925 15:25:43.975896 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:43.975943 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:43.976434 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:43.976703 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:43.976706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:43.977389 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:43.977589 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.977594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.978405 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.978406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.979245 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:43.979247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:43.979577 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:43.979578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:43.979680 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:43.980264 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:43.980274 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:43.981793 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:43.983265 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.983271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.983273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.983516 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.983739 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:25:43.983739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:25:43.983740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:25:43.983843 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:25:43.984069 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:43.984072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:43.984188 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 15:25:43.984269 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:25:43.984445 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":129,"threads_started":1}
I20260925 15:25:43.984833 1677 external_mini_cluster.cc:1428] Setting key 91cf66d3363e22b31e807f82af5cdf70
I20260925 15:25:43.985349 10327 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36453
I20260925 15:25:43.985481 10327 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:43.985677 10327 heartbeater.cc:511] Master 127.1.163.126:36453 requested a full tablet report, sending...
I20260925 15:25:43.986054 9867 ts_manager.cc:194] Registered new tserver with Master: 04c768a7f93d4323a3e8bfd047d3d787 (127.1.163.67:36703)
I20260925 15:25:43.986475 9867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:35731
I20260925 15:25:43.996810 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
Entry for principal oryx/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab.
WARNING: no policy specified for oryx/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
Entry for principal oryx/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab.
WARNING: no policy specified for oryx/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
Entry for principal oryx/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab.
I20260925 15:25:44.085156 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9930
I20260925 15:25:44.090150 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10064
I20260925 15:25:44.094955 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10198
I20260925 15:25:44.099390 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 9836
I20260925 15:25:44.104941 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36453
--webserver_interface=127.1.163.126
--webserver_port=41059
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:25:44.189725 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:44.189957 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:44.189980 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:44.191108 10336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:44.191145 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:44.191159 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:44.191171 10336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:44.191183 10336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:44.192703 10336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=41059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:44.192920 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:44.193102 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:44.197983 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.197989 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.197992 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:44.198390 10336 server_base.cc:1034] running on GCE node
I20260925 15:25:44.198643 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:44.198997 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:44.200145 10336 hybrid_clock.cc:648] HybridClock initialized: now 1790349944200122 us; error 34 us; skew 500 ppm
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:44.202885 10336 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:25:44.203394 10336 webserver.cc:460] Webserver started at http://127.1.163.126:41059/ using document root <none> and password file <none>
I20260925 15:25:44.203626 10336 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:44.203701 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:44.205195 10336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:44.205857 10351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.206033 10336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.206140 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "80bb8c712030458192d0dedbae7350e4"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "17279c705a00749a363baa7786b5310e"
server_key_iv: "481cc9aa9f5c7a51357f67c6e5f3178e"
server_key_version: "encryptionkey@0"
I20260925 15:25:44.206549 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:44.220028 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:44.222360 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:44.222539 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:44.226805 10336 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36453
I20260925 15:25:44.226841 10403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36453 every 8 connection(s)
I20260925 15:25:44.227262 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:44.228789 10404 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:25:44.229486 10404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Bootstrap starting.
I20260925 15:25:44.229954 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10336
I20260925 15:25:44.230309 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:33421
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:36453
--webserver_port=32907
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:25:44.237689 10404 log.cc:824] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:44.239481 10404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:25:44.239785 10404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Bootstrap complete.
I20260925 15:25:44.242111 10404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:44.242373 10404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bb8c712030458192d0dedbae7350e4, State: Initialized, Role: FOLLOWER
I20260925 15:25:44.242583 10404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:44.242682 10404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:44.242735 10404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:44.242784 10404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:25:44.243830 10404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:44.243957 10404 leader_election.cc:302] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80bb8c712030458192d0dedbae7350e4; no voters:
I20260925 15:25:44.244151 10404 leader_election.cc:288] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:25:44.244236 10408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:25:44.244427 10404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:44.244462 10408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [term 2 LEADER]: Becoming Leader. State: Replica: 80bb8c712030458192d0dedbae7350e4, State: Running, Role: LEADER
I20260925 15:25:44.244695 10408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } }
I20260925 15:25:44.245016 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80bb8c712030458192d0dedbae7350e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } } }
I20260925 15:25:44.245090 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:44.245242 10410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bb8c712030458192d0dedbae7350e4. Latest consensus state: current_term: 2 leader_uuid: "80bb8c712030458192d0dedbae7350e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bb8c712030458192d0dedbae7350e4" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36453 } } }
I20260925 15:25:44.245292 10410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:44.246297 10424 catalog_manager.cc:1259] Loaded cluster ID: 0944c7a27a964552b227cc5baac4d538
I20260925 15:25:44.246345 10424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: loading cluster ID for follower catalog manager: success
I20260925 15:25:44.247581 10424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: acquiring CA information for follower catalog manager: success
I20260925 15:25:44.247922 10424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:25:44.248013 10417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:44.248219 10417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:44.248302 10417 catalog_manager.cc:1259] Loaded cluster ID: 0944c7a27a964552b227cc5baac4d538
I20260925 15:25:44.248333 10417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:44.248519 10417 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:44.248656 10417 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80bb8c712030458192d0dedbae7350e4: Loaded TSK: 0
I20260925 15:25:44.248782 10417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:25:44.316316 10406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:44.316591 10406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:44.316660 10406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:44.317919 10406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:44.317963 10406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:44.317992 10406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:44.319556 10406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.65.keytab
--principal=oryx/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:33421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=32907
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:44.319831 10406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:44.320103 10406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:44.325385 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.325389 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.325531 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:44.326067 10406 server_base.cc:1034] running on GCE node
I20260925 15:25:44.326321 10406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:44.326666 10406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:44.327862 10406 hybrid_clock.cc:648] HybridClock initialized: now 1790349944327835 us; error 44 us; skew 500 ppm
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349944, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:44.331082 10406 init.cc:377] Logged in from keytab as oryx/127.1.163.65@KRBTEST.COM (short username oryx)
I20260925 15:25:44.331607 10406 webserver.cc:460] Webserver started at http://127.1.163.65:32907/ using document root <none> and password file <none>
I20260925 15:25:44.331857 10406 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:44.331943 10406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:44.333539 10406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:44.334227 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.334427 10406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:25:44.334543 10406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "0c63b0670cc84dbdb64ddfc15882469c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "cb3415e35bb84186925472987c304cb4"
server_key_iv: "be97f3f73b3ed1342e22942013e5218c"
server_key_version: "encryptionkey@0"
I20260925 15:25:44.334911 10406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:44.357828 10406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:44.360363 10406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:44.360565 10406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:44.360862 10406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:44.361231 10406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:44.361289 10406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.361338 10406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:44.361366 10406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.368152 10406 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:33421
I20260925 15:25:44.368201 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:33421 every 8 connection(s)
I20260925 15:25:44.368646 10406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:44.375887 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10406
I20260925 15:25:44.376230 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:43421
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:36453
--webserver_port=36271
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.378961 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.369497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:43353)
0925 15:25:44.369716 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.369773 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.370339 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.370815 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.370818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.371495 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.371713 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.371718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.373426 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.373427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.374281 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.374285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.375161 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.375162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.375284 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.376041 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.376051 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.378718 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
W20260925 15:25:44.379213 10554 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.163.126:36453: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
W20260925 15:25:44.379312 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.369589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:43353 (local address 127.1.163.126:36453)
0925 15:25:44.369841 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.369850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.369899 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:25:44.370562 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.370569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.370623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.370766 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.371848 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.373296 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.374390 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.375065 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.375181 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.378864 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:43353: BlockingRecv error: recv got EOF from 127.1.163.65:43353 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.384240 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.379455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:54271)
0925 15:25:44.379514 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.379526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.379575 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.379725 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.379726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.379845 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.379893 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.379895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.380919 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.380920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.381420 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.381421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.381948 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.381949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.381994 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.382454 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.382459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.384185 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:25:44.384385 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.379553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:54271 (local address 127.1.163.126:36453)
0925 15:25:44.379599 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.379601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.379612 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:25:44.379637 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.379638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.379651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.379711 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.379982 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.380825 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.381514 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.381867 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.381947 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.384323 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:54271: BlockingRecv error: recv got EOF from 127.1.163.65:54271 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.388976 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.384696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:60113)
0925 15:25:44.384747 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.384758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.384795 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.384994 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.384995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.385122 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.385152 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.385153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.385778 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.385779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.386276 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.386277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.386654 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.386655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.386707 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.387155 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.387160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.388900 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:25:44.389132 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.384773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:60113 (local address 127.1.163.126:36453)
0925 15:25:44.384823 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.384825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.384839 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:25:44.384863 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.384863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.384873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.384945 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.385227 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.385698 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.386386 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.386578 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.386655 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.388981 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:60113: BlockingRecv error: recv got EOF from 127.1.163.65:60113 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:25:44.389150 10554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:25:44.460155 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:44.460374 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:44.460398 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:44.461563 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:44.461611 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:44.461638 10558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:25:44.463125 10558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.66.keytab
--principal=oryx/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:43421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=36271
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:44.463377 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:44.463630 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:44.468824 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.468840 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:44.468866 10558 server_base.cc:1034] running on GCE node
W20260925 15:25:44.468829 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:44.469254 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:44.469509 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:44.470654 10558 hybrid_clock.cc:648] HybridClock initialized: now 1790349944470643 us; error 30 us; skew 500 ppm
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349944, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:44.473199 10558 init.cc:377] Logged in from keytab as oryx/127.1.163.66@KRBTEST.COM (short username oryx)
I20260925 15:25:44.473682 10558 webserver.cc:460] Webserver started at http://127.1.163.66:36271/ using document root <none> and password file <none>
I20260925 15:25:44.473918 10558 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:44.474002 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:44.475410 10558 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:44.476035 10573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.476204 10558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:25:44.476302 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "e3dfb0b258c9467da23f1bbc1bee190c"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "ceb9f78cea66c2e888dbb0d18ba7f62d"
server_key_iv: "dacf4b73f768a5f772dcc24f9dc597bc"
server_key_version: "encryptionkey@0"
I20260925 15:25:44.476722 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:44.490193 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:44.492306 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:44.492486 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:44.492767 10558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:44.493088 10558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:44.493144 10558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.493189 10558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:44.493214 10558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.499166 10558 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:43421
I20260925 15:25:44.499210 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:43421 every 8 connection(s)
I20260925 15:25:44.499569 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:25:44.502439 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10558
I20260925 15:25:44.502766 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:36703
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:36453
--webserver_port=40595
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:25:44.507800 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.500311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:44699 (local address 127.1.163.126:36453)
0925 15:25:44.500376 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.500378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.500645 (+ 267us) server_negotiation.cc:407] Connection header received
0925 15:25:44.501189 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.501191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.501208 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.501266 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.502233 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.502825 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.503958 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.504157 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.504240 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.507744 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:44699: BlockingRecv error: recv got EOF from 127.1.163.66:44699 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:25:44.507865 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.500301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:44699)
0925 15:25:44.500489 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.500526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.501078 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.501340 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.501344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.501926 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.502126 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.502130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.502962 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.502963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.503782 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.503784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.504247 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.504249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.504380 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.504996 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.505005 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.507662 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
W20260925 15:25:44.508168 10687 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.163.126:36453: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.513862 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.508454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:33017)
0925 15:25:44.508561 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.508574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.508618 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.508780 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.508781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.508950 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.509015 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.509017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.509694 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.509695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.510604 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.510608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.511008 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.511009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.511091 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.511521 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.511527 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.513754 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 15:25:44.513891 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.508532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:33017 (local address 127.1.163.126:36453)
0925 15:25:44.508612 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.508613 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.508621 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:44.508699 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.508700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.508708 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.508757 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.509098 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.509587 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.510709 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.510931 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.510985 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.513837 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:33017: BlockingRecv error: recv got EOF from 127.1.163.66:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.518733 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.514223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:56617)
0925 15:25:44.514277 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.514290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.514334 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.514487 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.514488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.514594 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.514623 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.514624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.515525 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.515526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.515998 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.515999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.516440 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.516441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.516477 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.516990 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.516994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.518647 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:25:44.518810 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.514300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:56617 (local address 127.1.163.126:36453)
0925 15:25:44.514361 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.514363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.514393 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:25:44.514417 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.514418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.514428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.514477 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.514695 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.515442 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.516070 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.516370 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.516437 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.518736 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:56617: BlockingRecv error: recv got EOF from 127.1.163.66:56617 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 15:25:44.518957 10687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:25:44.589603 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:44.589885 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:44.589928 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:44.591169 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:44.591243 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:44.591290 10690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:25:44.592829 10690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/krb5kdc/oryx_127.1.163.67.keytab
--principal=oryx/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:36703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=40595
--tserver_master_addrs=127.1.163.126:36453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:44.593102 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:44.593355 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:44.598495 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.598516 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:44.598510 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:44.598688 10690 server_base.cc:1034] running on GCE node
I20260925 15:25:44.598994 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:44.599242 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:44.600415 10690 hybrid_clock.cc:648] HybridClock initialized: now 1790349944600402 us; error 29 us; skew 500 ppm
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349944, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:25:44.603103 10690 init.cc:377] Logged in from keytab as oryx/127.1.163.67@KRBTEST.COM (short username oryx)
I20260925 15:25:44.603513 10690 webserver.cc:460] Webserver started at http://127.1.163.67:40595/ using document root <none> and password file <none>
I20260925 15:25:44.603744 10690 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:44.603826 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:44.605329 10690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:25:44.605944 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.606204 10690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:44.606277 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "04c768a7f93d4323a3e8bfd047d3d787"
format_stamp: "Formatted at 2026-09-25 15:25:43 on dist-test-slave-hvv5"
server_key: "bbe54cf91c14089934aa55a88576f55a"
server_key_iv: "8091dd2ad64710ee3db47d9953b341a9"
server_key_version: "encryptionkey@0"
I20260925 15:25:44.606673 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:44.631271 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:44.633862 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:44.633991 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:44.634294 10690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:44.634620 10690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:44.634676 10690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.634712 10690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:44.634739 10690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:44.640887 10690 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:36703
I20260925 15:25:44.640928 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:36703 every 8 connection(s)
I20260925 15:25:44.641351 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.648794 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10690
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:25:44.650071 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.642182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:46945 (local address 127.1.163.126:36453)
0925 15:25:44.642252 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.642254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.642483 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:25:44.643086 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.643087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.643101 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.643153 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.644226 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.644916 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.646011 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.646229 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.646325 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.650008 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:46945: BlockingRecv error: recv got EOF from 127.1.163.67:46945 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:25:44.650154 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.642171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:46945)
0925 15:25:44.642357 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.642394 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.642961 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.643224 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.643228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.643909 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.644115 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.644120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.645008 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.645009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.645915 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.645917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.646335 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.646337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.646475 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.647049 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.647058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.649936 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
W20260925 15:25:44.650411 10819 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.163.126:36453: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.655710 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.650645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:40071)
0925 15:25:44.650694 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.650704 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.650737 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.650902 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.650903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.651056 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.651107 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.651109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.652046 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.652046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.652796 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.652799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.653241 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.653242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.653321 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.653765 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.653769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.655619 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:25:44.655767 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.650693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40071 (local address 127.1.163.126:36453)
0925 15:25:44.650728 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.650730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.650739 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:44.650789 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.650790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.650797 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.650846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.651186 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.651957 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.652885 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.653144 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.653198 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.655692 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:40071: BlockingRecv error: recv got EOF from 127.1.163.67:40071 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:25:44.657704 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.649437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.1.163.126:36453)
0925 15:25:44.649557 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.649559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.649567 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:44.649592 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.649593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.649605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.649683 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.650373 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.651189 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.652280 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.652641 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.652713 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.654173 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:44.654180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:44.654180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:25:44.654194 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:44.656198 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:44.656600 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:44.656604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:44.656605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:44.656709 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:25:44.656924 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:25:44.656925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:25:44.656926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:25:44.657164 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:44.657329 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:44.657443 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:25:44.657571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:44 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:44.660205 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:44.656016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:52297)
0925 15:25:44.656055 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:44.656067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:44.656106 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:44.656240 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:44.656241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:44.656355 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:44.656384 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.656386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.657079 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.657080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.657574 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:44.657576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.658019 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.658020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:44.658084 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.658459 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:44.658463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:44.660154 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 15:25:44.660384 10557 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:44.656070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:52297 (local address 127.1.163.126:36453)
0925 15:25:44.656105 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:44.656107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:44.656115 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:44.656160 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:44.656160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:44.656170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:44.656224 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:44.656459 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.656998 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.657679 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:44.657942 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:44.658011 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:44.660317 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:52297: BlockingRecv error: recv got EOF from 127.1.163.67:52297 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:25:44.660385 10819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:45.395313 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:45.389721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:40127)
0925 15:25:45.389864 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:45.389878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:45.389977 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:45.390247 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:45.390249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:45.390450 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:45.390505 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.390507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.391364 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.391365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.391953 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.391955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.392345 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.392346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.392400 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.392933 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:45.392938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:45.395217 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260925 15:25:45.395632 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:45.389806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:40127 (local address 127.1.163.126:36453)
0925 15:25:45.389988 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:45.389991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:45.390001 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:45.390090 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:45.390090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:45.390103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:45.390181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:45.390580 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.391273 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.392051 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.392255 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.392324 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.395362 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:40127: BlockingRecv error: recv got EOF from 127.1.163.65:40127 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:45.525533 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:45.519511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:45071)
0925 15:25:45.519805 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:45.519816 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:45.519908 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:45.520078 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:45.520079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:45.520275 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:45.520322 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.520324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.521064 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.521066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.521621 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.521623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.522051 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.522052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.522119 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.522670 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:45.522676 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:45.525455 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
W20260925 15:25:45.525856 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:45.519643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:45071 (local address 127.1.163.126:36453)
0925 15:25:45.519738 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:45.519741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:45.519887 (+ 146us) server_negotiation.cc:407] Connection header received
0925 15:25:45.519979 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:45.519980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:45.519994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:45.520070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:45.520404 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.520976 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.521742 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.521942 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.522035 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.525809 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:45071: BlockingRecv error: recv got EOF from 127.1.163.66:45071 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:45 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:45.665597 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:45.660798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:33773)
0925 15:25:45.660962 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:45.660976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:45.661044 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:45.661188 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:45.661189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:45.661372 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:45.661418 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.661419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.662100 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.662101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.662655 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:45.662656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.663025 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.663026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:45.663082 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.663562 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:45.663566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:45.665521 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
W20260925 15:25:45.665699 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:45.660921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:33773 (local address 127.1.163.126:36453)
0925 15:25:45.660973 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:45.660976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:45.661027 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:25:45.661099 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:45.661100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:45.661114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:45.661170 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:45.661501 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.662000 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.662753 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:45.662951 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:45.663028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:45.665626 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:33773: BlockingRecv error: recv got EOF from 127.1.163.67:33773 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:46.401259 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:46.395950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:35951)
0925 15:25:46.396136 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:46.396147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:46.396225 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:46.396362 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:46.396363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:46.396585 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:46.396645 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.396647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.397372 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.397374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.397967 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.397969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.398388 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.398389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.398448 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.398976 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:46.398982 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:46.401176 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
W20260925 15:25:46.401379 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:46.396023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:35951 (local address 127.1.163.126:36453)
0925 15:25:46.396127 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:46.396129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:46.396238 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:25:46.396263 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:46.396263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:46.396274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:46.396333 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:46.396753 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.397266 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.398060 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.398289 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.398388 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.401300 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:35951: BlockingRecv error: recv got EOF from 127.1.163.65:35951 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:46.531184 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:46.526165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:59965)
0925 15:25:46.526297 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:46.526309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:46.526384 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:46.526568 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:46.526569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:46.526730 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:46.526781 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.526782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.527502 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.527504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.528230 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.528234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.528839 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.528841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.528905 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.529416 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:46.529420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:46.531127 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
W20260925 15:25:46.531275 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:46.526250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:59965 (local address 127.1.163.126:36453)
0925 15:25:46.526361 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:46.526363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:46.526376 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:25:46.526485 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:46.526486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:46.526498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:46.526554 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:46.526898 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.527404 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.528381 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.528760 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.528842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.531227 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:59965: BlockingRecv error: recv got EOF from 127.1.163.66:59965 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:46 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:46.671509 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:46.666242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:60911)
0925 15:25:46.666502 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:46.666521 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:46.666582 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:46.666805 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:46.666806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:46.666976 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:46.667024 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.667026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.667785 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.667786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.668318 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:46.668320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.668776 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.668777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:46.668825 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.669364 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:46.669369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:46.671405 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":209,"threads_started":1}
W20260925 15:25:46.671561 10829 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:46.666472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:60911 (local address 127.1.163.126:36453)
0925 15:25:46.666529 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:46.666532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:46.666569 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:25:46.666649 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:46.666650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:46.666662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:46.666708 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:46.667154 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.667651 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.668403 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:46.668664 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:46.668733 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:46.671494 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:60911: BlockingRecv error: recv got EOF from 127.1.163.67:60911 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:47.406914 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:47.401859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:47609)
0925 15:25:47.402018 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:47.402029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:47.402102 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:47.402266 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:47.402267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:47.402476 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:47.402541 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.402543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.403462 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.403464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.404009 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.404011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.404332 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.404333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.404380 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.404898 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:47.404903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:47.406851 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
W20260925 15:25:47.407069 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:47.401942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:47609 (local address 127.1.163.126:36453)
0925 15:25:47.402048 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:47.402050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:47.402100 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:25:47.402151 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:47.402152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:47.402161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:47.402223 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:47.402621 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.403368 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.404079 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.404263 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.404315 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.406967 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:47609: BlockingRecv error: recv got EOF from 127.1.163.65:47609 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:47.536403 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:47.531855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:47789)
0925 15:25:47.532017 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:47.532027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:47.532085 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:47.532228 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:47.532228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:47.532390 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:47.532440 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.532442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.533087 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.533088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.533617 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.533618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.533936 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.533936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.533975 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.534439 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:47.534444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:47.536345 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260925 15:25:47.536538 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:47.531969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:47789 (local address 127.1.163.126:36453)
0925 15:25:47.532035 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:47.532038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:47.532087 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:25:47.532133 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:47.532134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:47.532145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:47.532209 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:47.532505 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.533019 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.533691 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.533871 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.533923 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.536454 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:47789: BlockingRecv error: recv got EOF from 127.1.163.66:47789 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:47 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:47.676888 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:47.672143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:48235)
0925 15:25:47.672306 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:47.672318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:47.672384 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:47.672630 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:47.672631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:47.672823 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:47.672873 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.672875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.673557 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.673557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.674099 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:47.674100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.674428 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.674429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:47.674472 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.675045 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:47.675049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:47.676826 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
W20260925 15:25:47.677049 10833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:47.672202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:48235 (local address 127.1.163.126:36453)
0925 15:25:47.672273 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:47.672275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:47.672406 (+ 131us) server_negotiation.cc:407] Connection header received
0925 15:25:47.672437 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:47.672438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:47.672452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:47.672512 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:47.672976 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.673481 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.674174 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:47.674360 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:47.674411 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:47.676954 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:48235: BlockingRecv error: recv got EOF from 127.1.163.67:48235 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:48.416917 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:48.407528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:56189)
0925 15:25:48.407724 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:48.407739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:48.407815 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:48.407956 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:48.407956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:48.408121 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:48.408172 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.408173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.409002 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.409004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.409668 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.409671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.410186 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.410187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.410245 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.414902 (+ 4657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:48.414909 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:48.416853 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
W20260925 15:25:48.417096 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:48.407603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:56189 (local address 127.1.163.126:36453)
0925 15:25:48.407758 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:48.407760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:48.407798 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:25:48.407864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:48.407864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:48.407875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:48.407936 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:48.408242 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.408920 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.409774 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.410060 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.410192 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.417004 (+ 6812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:56189: BlockingRecv error: recv got EOF from 127.1.163.65:56189 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:48.542289 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:48.537030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:45305)
0925 15:25:48.537199 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:48.537211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:48.537260 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:48.537451 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:48.537453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:48.537720 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:48.537771 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.537773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.538548 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.538549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.539241 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.539244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.539688 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.539689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.539753 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.540325 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:48.540330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:48.542185 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260925 15:25:48.542357 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:48.537158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:45305 (local address 127.1.163.126:36453)
0925 15:25:48.537234 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:48.537236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:48.537272 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:25:48.537333 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:48.537334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:48.537347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:48.537421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:48.537866 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.538436 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.539350 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.539587 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.539687 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.542283 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:45305: BlockingRecv error: recv got EOF from 127.1.163.66:45305 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:48 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:48.682849 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:48.677457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:39529)
0925 15:25:48.677676 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:48.677692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:48.677780 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:48.678009 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:48.678011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:48.678202 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:48.678278 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.678281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.679134 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.679135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.679735 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:48.679738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.680148 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.680149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:48.680198 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.680813 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:48.680817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:48.682786 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
W20260925 15:25:48.682989 10837 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:48.677615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:39529 (local address 127.1.163.126:36453)
0925 15:25:48.677709 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:48.677712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:48.677760 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:25:48.677906 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:48.677907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:48.677919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:48.677997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:48.678415 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.679052 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.679837 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:48.680062 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:48.680135 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:48.682908 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:39529: BlockingRecv error: recv got EOF from 127.1.163.67:39529 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:49.422938 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:49.417471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:44255)
0925 15:25:49.417637 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:49.417649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:49.417709 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:49.417966 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:49.417968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:49.418139 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:49.418190 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.418192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.418979 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.418980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.419583 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.419585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.420057 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.420058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.420109 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.420871 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:49.420876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:49.422877 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
W20260925 15:25:49.423091 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:49.417605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:44255 (local address 127.1.163.126:36453)
0925 15:25:49.417772 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:49.417774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:49.417784 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:49.417812 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:49.417813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:49.417827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:49.417914 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:49.418314 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.418862 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.419733 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.419967 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.420027 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.422990 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:44255: BlockingRecv error: recv got EOF from 127.1.163.65:44255 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:49.548586 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:49.542849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:53009)
0925 15:25:49.543113 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:49.543129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:49.543225 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:49.543400 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:49.543401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:49.543602 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:49.543652 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.543653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.544595 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.544596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.545443 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.545445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.545865 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.545866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.545928 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.546485 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:49.546490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:49.548469 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
W20260925 15:25:49.548781 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:49.543000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:53009 (local address 127.1.163.126:36453)
0925 15:25:49.543075 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:49.543078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:49.543204 (+ 126us) server_negotiation.cc:407] Connection header received
0925 15:25:49.543296 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:49.543298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:49.543312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:49.543395 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:49.543777 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.544472 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.545533 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.545771 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.545865 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.548713 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:53009: BlockingRecv error: recv got EOF from 127.1.163.66:53009 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:49 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:49.688854 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:49.683441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:40187)
0925 15:25:49.683594 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:49.683604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:49.683665 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:49.683885 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:49.683886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:49.684072 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:49.684122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.684124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.684878 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.684879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.685404 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:49.685405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.685996 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.685998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:49.686048 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.686676 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:49.686682 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:49.688785 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260925 15:25:49.689033 10841 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:49.683516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40187 (local address 127.1.163.126:36453)
0925 15:25:49.683613 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:49.683615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:49.683654 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:25:49.683753 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:49.683754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:49.683769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:49.683836 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:49.684227 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.684755 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.685503 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:49.685705 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:49.685812 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:49.688923 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:40187: BlockingRecv error: recv got EOF from 127.1.163.67:40187 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:50.428844 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:50.423568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:43919)
0925 15:25:50.423714 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:50.423733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:50.423786 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:50.424003 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:50.424004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:50.424168 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:50.424217 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.424219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.424959 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.424960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.425509 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.425511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.425898 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.425899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.425955 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.426525 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:50.426530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:50.428782 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260925 15:25:50.429033 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:50.423666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:43919 (local address 127.1.163.126:36453)
0925 15:25:50.423852 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:50.423854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:50.423864 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:50.423898 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:50.423899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:50.423910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:50.423985 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:50.424329 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.424866 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.425593 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.425791 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.425856 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.428943 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:43919: BlockingRecv error: recv got EOF from 127.1.163.65:43919 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:50.554296 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:50.549138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:46325)
0925 15:25:50.549324 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:50.549349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:50.549426 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:50.549696 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:50.549697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:50.549932 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:50.549982 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.549984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.550670 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.550670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.551206 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.551207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.551572 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.551573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.551614 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.552124 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:50.552130 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:50.554230 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
W20260925 15:25:50.554489 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:50.549187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:46325 (local address 127.1.163.126:36453)
0925 15:25:50.549254 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:50.549256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:50.549452 (+ 196us) server_negotiation.cc:407] Connection header received
0925 15:25:50.549486 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:50.549487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:50.549502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:50.549577 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:50.550078 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.550598 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.551303 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.551497 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.551565 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.554414 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:46325: BlockingRecv error: recv got EOF from 127.1.163.66:46325 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:50 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:50.694406 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:50.689462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:59807)
0925 15:25:50.689616 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:50.689629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:50.689688 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:50.689935 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:50.689936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:50.690120 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:50.690168 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.690169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.690850 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.690851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.691397 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:50.691398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.691822 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.691824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:50.691894 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.692452 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:50.692471 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:50.694315 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
W20260925 15:25:50.694445 10845 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:50.689574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:59807 (local address 127.1.163.126:36453)
0925 15:25:50.689661 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:50.689664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:50.689690 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:25:50.689822 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:50.689823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:50.689845 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:50.689925 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:50.690255 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.690754 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.691524 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:50.691716 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:50.691821 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:50.694383 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:59807: BlockingRecv error: recv got EOF from 127.1.163.67:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:51.434433 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:51.429474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:41877)
0925 15:25:51.429618 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:51.429639 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:51.429696 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:51.429906 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:51.429907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:51.430070 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:51.430120 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.430122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.430838 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.430840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.431367 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.431369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.431717 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.431717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.431757 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.432293 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:51.432298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:51.434371 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260925 15:25:51.434609 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:51.429561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:41877 (local address 127.1.163.126:36453)
0925 15:25:51.429772 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:51.429774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:51.429784 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:25:51.429808 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:51.429808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:51.429820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:51.429887 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:51.430217 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.430726 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.431453 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.431643 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.431708 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.434508 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:41877: BlockingRecv error: recv got EOF from 127.1.163.65:41877 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":148,"threads_started":1}
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:51.560340 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:51.554915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:60545)
0925 15:25:51.555088 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:51.555101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:51.555174 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:51.555376 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:51.555378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:51.555562 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:51.555618 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.555620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.556318 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.556319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.556895 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.556897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.557347 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.557349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.557412 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.558152 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:51.558166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:51.560279 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
W20260925 15:25:51.560520 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:51.555038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:60545 (local address 127.1.163.126:36453)
0925 15:25:51.555108 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:51.555109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:51.555166 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:25:51.555239 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:51.555240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:51.555251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:51.555321 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:51.555736 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.556226 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.556978 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.557271 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.557325 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.560447 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:60545: BlockingRecv error: recv got EOF from 127.1.163.66:60545 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:51 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:25:51.699766 10849 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:51.695076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40553 (local address 127.1.163.126:36453)
0925 15:25:51.695147 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:51.695149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:51.695274 (+ 125us) server_negotiation.cc:407] Connection header received
0925 15:25:51.695323 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:51.695323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:51.695344 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:51.695391 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:51.695670 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.696193 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.697056 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.697233 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.697280 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.699713 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:40553: BlockingRecv error: recv got EOF from 127.1.163.67:40553 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:25:51.699815 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:51.695013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:40553)
0925 15:25:51.695194 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:51.695206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:51.695275 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:51.695405 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:51.695406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:51.695564 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:51.695614 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.695616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.696290 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.696291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.696973 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:51.696975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:51.697298 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:51.697299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:51.697344 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:51.697786 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:51.697791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:51.699740 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:52.439765 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:52.434950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:50979)
0925 15:25:52.435087 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:52.435099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:52.435156 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:52.435369 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:52.435370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:52.435530 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:52.435579 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.435581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.436241 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.436242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.436816 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.436818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.437190 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.437191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.437262 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.437831 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:52.437837 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:52.439692 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
W20260925 15:25:52.439894 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:52.435108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:50979 (local address 127.1.163.126:36453)
0925 15:25:52.435228 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:52.435230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:52.435238 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:52.435269 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:52.435270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:52.435279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:52.435341 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:52.435651 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.436160 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.436893 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.437090 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.437147 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.439791 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:50979: BlockingRecv error: recv got EOF from 127.1.163.65:50979 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:52.566015 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:52.560930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:43781)
0925 15:25:52.561087 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:52.561098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:52.561167 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:52.561334 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:52.561335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:52.561497 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:52.561546 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.561547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.562355 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.562356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.562930 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.562931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.563319 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.563320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.563383 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.563934 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:52.563940 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:52.565955 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":56,"threads_started":1}
W20260925 15:25:52.566159 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:52.561028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:43781 (local address 127.1.163.126:36453)
0925 15:25:52.561098 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:52.561100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:52.561169 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:25:52.561217 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:52.561218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:52.561228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:52.561279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:52.561621 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.562264 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.563007 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.563203 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.563254 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.566069 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:43781: BlockingRecv error: recv got EOF from 127.1.163.66:43781 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:52 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:52.705399 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:52.700408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:34731)
0925 15:25:52.700613 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:52.700639 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:52.700708 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:52.700906 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:52.700908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:52.701073 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:52.701123 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.701124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.701823 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.701824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.702370 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:52.702372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.702765 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.702767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:52.702826 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.703336 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:52.703341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:52.705300 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
W20260925 15:25:52.705451 10853 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:52.700580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34731 (local address 127.1.163.126:36453)
0925 15:25:52.700658 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:52.700660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:52.700694 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:25:52.700760 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:52.700760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:52.700778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:52.700850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:52.701204 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.701726 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.702463 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:52.702670 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:52.702738 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:52.705366 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:34731: BlockingRecv error: recv got EOF from 127.1.163.67:34731 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:53.449954 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:53.440280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:32849)
0925 15:25:53.440443 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:53.440457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:53.440560 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:53.440746 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:53.440747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:53.440933 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:53.440985 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.440987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.441673 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.441674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.442258 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.442260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.442630 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.442632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.442682 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.447903 (+ 5221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:53.447909 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:53.449862 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
W20260925 15:25:53.450105 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:53.440408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:32849 (local address 127.1.163.126:36453)
0925 15:25:53.440584 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:53.440587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:53.440602 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:25:53.440629 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:53.440630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:53.440642 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:53.440704 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:53.441058 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.441575 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.442338 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.442534 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.442589 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.450015 (+ 7426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:32849: BlockingRecv error: recv got EOF from 127.1.163.65:32849 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:53.571610 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:53.566579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:41945)
0925 15:25:53.566742 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:53.566762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:53.566827 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:53.567002 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:53.567004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:53.567177 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:53.567230 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.567231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.567957 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.567959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.568492 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.568493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.568969 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.568970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.569022 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.569579 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:53.569583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:53.571542 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
W20260925 15:25:53.571784 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:53.566694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:41945 (local address 127.1.163.126:36453)
0925 15:25:53.566771 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:53.566773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:53.566806 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:25:53.566887 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:53.566887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:53.566898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:53.566964 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:53.567325 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.567879 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.568626 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.568872 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.568942 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.571703 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:41945: BlockingRecv error: recv got EOF from 127.1.163.66:41945 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:53 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:53.710922 10859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:53.706128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:38723)
0925 15:25:53.706293 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:53.706303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:53.706358 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:53.706509 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:53.706510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:53.706670 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:53.706721 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.706723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.707376 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.707376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.707900 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:53.707902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.708242 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.708243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:53.708287 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.708804 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:53.708809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:53.710854 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
W20260925 15:25:53.711055 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:53.706206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:38723 (local address 127.1.163.126:36453)
0925 15:25:53.706291 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:53.706293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:53.706349 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:25:53.706423 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:53.706424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:53.706436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:53.706490 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:53.706793 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.707304 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.707988 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:53.708178 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:53.708231 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:53.710951 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:38723: BlockingRecv error: recv got EOF from 127.1.163.67:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:54.455540 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:54.450477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:36505)
0925 15:25:54.450675 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:54.450691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:54.450770 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:54.450955 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:54.450956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:54.451176 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:54.451232 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.451233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.451879 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.451880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.452465 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.452467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.452952 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.452953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.453005 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.453501 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:54.453506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:54.455441 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
W20260925 15:25:54.455586 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:54.450602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:36505 (local address 127.1.163.126:36453)
0925 15:25:54.450758 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:54.450760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:54.450769 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:54.450844 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:54.450844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:54.450854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:54.450929 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:54.451300 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.451801 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.452582 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.452882 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.452938 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.455516 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:36505: BlockingRecv error: recv got EOF from 127.1.163.65:36505 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:54.577479 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:54.572189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:39405)
0925 15:25:54.572362 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:54.572379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:54.572441 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:54.572642 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:54.572643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:54.572832 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:54.572883 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.572885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.573585 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.573586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.574113 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.574115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.574540 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.574541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.574602 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.575167 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:54.575172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:54.577294 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
W20260925 15:25:54.577704 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:54.572322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:39405 (local address 127.1.163.126:36453)
0925 15:25:54.572396 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:54.572398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:54.572432 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:25:54.572511 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:54.572511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:54.572522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:54.572591 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:54.572980 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.573491 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.574220 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.574439 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.574508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.577644 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:39405: BlockingRecv error: recv got EOF from 127.1.163.66:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:54 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:54.716459 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:54.711478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:44835)
0925 15:25:54.711629 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:54.711641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:54.711716 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:54.711895 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:54.711896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:54.712124 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:54.712188 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.712190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.712874 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.712875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.713414 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:54.713415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.713976 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.713977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:54.714030 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.714491 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:54.714496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:54.716390 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260925 15:25:54.716619 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:54.711585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:44835 (local address 127.1.163.126:36453)
0925 15:25:54.711631 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:54.711634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:54.711726 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:25:54.711769 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:54.711770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:54.711805 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:54.711863 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:54.712274 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.712802 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.713641 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:54.713881 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:54.713949 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:54.716501 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:44835: BlockingRecv error: recv got EOF from 127.1.163.67:44835 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:55.461280 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:55.456191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:51867)
0925 15:25:55.456358 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:55.456368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:55.456417 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:55.456615 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:55.456615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:55.456785 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:55.456835 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.456837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.457562 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.457563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.458154 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.458157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.458543 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.458544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.458605 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.459124 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:55.459129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:55.461210 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
W20260925 15:25:55.461450 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:55.456314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:51867 (local address 127.1.163.126:36453)
0925 15:25:55.456471 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:55.456472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:55.456480 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:55.456501 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:55.456502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:55.456512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:55.456594 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:55.456953 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.457476 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.458263 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.458465 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.458531 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.461346 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:51867: BlockingRecv error: recv got EOF from 127.1.163.65:51867 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:55.583163 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:55.577960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:48925)
0925 15:25:55.578146 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:55.578157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:55.578248 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:55.578416 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:55.578417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:55.578607 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:55.578654 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.578655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.579416 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.579417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.579999 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.580001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.580433 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.580434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.580502 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.581117 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:55.581123 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:55.583094 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
W20260925 15:25:55.583344 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:55.578026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:48925 (local address 127.1.163.126:36453)
0925 15:25:55.578123 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:55.578125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:55.578227 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:25:55.578316 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:55.578317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:55.578335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:55.578398 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:55.578740 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.579324 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.580105 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.580348 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.580423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.583270 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:48925: BlockingRecv error: recv got EOF from 127.1.163.66:48925 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:55 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:55.722278 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:55.717066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:36213)
0925 15:25:55.717213 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:55.717224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:55.717284 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:55.717425 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:55.717426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:55.717596 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:55.717644 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.717645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.718311 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.718312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.718860 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:55.718863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.719226 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.719227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:55.719282 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.719880 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:55.719885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:55.722167 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260925 15:25:55.722321 10865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:55.717164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:36213 (local address 127.1.163.126:36453)
0925 15:25:55.717234 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:55.717236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:55.717280 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:25:55.717335 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:55.717336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:55.717347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:55.717408 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:55.717709 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.718238 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.718940 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:55.719123 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:55.719184 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:55.722257 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:36213: BlockingRecv error: recv got EOF from 127.1.163.67:36213 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:56.466768 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:56.461888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:37769)
0925 15:25:56.462060 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:56.462080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:56.462154 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:56.462283 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:56.462283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:56.462453 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:56.462502 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.462503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.463177 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.463178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.463762 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.463764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.464334 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.464335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.464387 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.464889 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:56.464895 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:56.466707 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
W20260925 15:25:56.466902 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:56.461997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:37769 (local address 127.1.163.126:36453)
0925 15:25:56.462148 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:56.462150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:56.462158 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:25:56.462206 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:56.462206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:56.462217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:56.462272 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:56.462568 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.463089 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.463859 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.464239 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.464303 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.466785 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:37769: BlockingRecv error: recv got EOF from 127.1.163.65:37769 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:56.588526 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:56.583735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:58929)
0925 15:25:56.583937 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:56.583950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:56.584053 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:56.584219 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:56.584220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:56.584415 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:56.584461 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.584463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.585217 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.585218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.585749 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.585751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.586136 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.586137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.586199 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.586670 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:56.586674 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:56.588463 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":114,"threads_started":1}
W20260925 15:25:56.588716 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:56.583868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:58929 (local address 127.1.163.126:36453)
0925 15:25:56.583941 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:56.583944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:56.584030 (+ 86us) server_negotiation.cc:407] Connection header received
0925 15:25:56.584124 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:56.584125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:56.584144 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:56.584211 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:56.584569 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.585111 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.585839 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.586045 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.586134 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.588633 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:58929: BlockingRecv error: recv got EOF from 127.1.163.66:58929 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:56 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:56.727734 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:56.722831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:34247)
0925 15:25:56.723038 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:56.723052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:56.723146 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:56.723350 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:56.723351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:56.723548 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:56.723596 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.723598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.724298 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.724299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.724925 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:56.724927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.725306 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.725307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:56.725370 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.725821 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:56.725825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:56.727636 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
W20260925 15:25:56.727794 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:56.722936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34247 (local address 127.1.163.126:36453)
0925 15:25:56.723023 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:56.723026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:56.723127 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:25:56.723232 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:56.723233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:56.723247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:56.723331 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:56.723685 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.724202 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.725012 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:56.725219 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:56.725305 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:56.727729 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:34247: BlockingRecv error: recv got EOF from 127.1.163.67:34247 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:57.472311 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:57.467351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:52791)
0925 15:25:57.467512 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:57.467524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:57.467593 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:57.467827 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:57.467828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:57.467997 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:57.468047 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.468048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.468984 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.468985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.469550 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.469552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.469899 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.469900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.469949 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.470392 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:57.470397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:57.472251 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
W20260925 15:25:57.472440 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:57.467537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:52791 (local address 127.1.163.126:36453)
0925 15:25:57.467656 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:57.467658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:57.467667 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:57.467700 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:57.467700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:57.467711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:57.467772 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:57.468303 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.468910 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.469624 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.469817 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.469886 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.472359 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:52791: BlockingRecv error: recv got EOF from 127.1.163.65:52791 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:57.594030 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:57.589236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:45921)
0925 15:25:57.589412 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:57.589423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:57.589494 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:57.589636 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:57.589637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:57.589806 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:57.589861 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.589863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.590548 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.590548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.591070 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.591071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.591433 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.591434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.591474 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.592006 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:57.592012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:57.593927 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":113,"threads_started":1}
W20260925 15:25:57.594086 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:57.589358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:45921 (local address 127.1.163.126:36453)
0925 15:25:57.589427 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:57.589429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:57.589495 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:25:57.589559 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:57.589559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:57.589571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:57.589616 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:57.589956 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.590469 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.591154 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.591360 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.591419 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.594025 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:45921: BlockingRecv error: recv got EOF from 127.1.163.66:45921 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:57 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:57.733449 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:57.728344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:52153)
0925 15:25:57.728530 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:57.728572 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:57.728630 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:57.728775 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:57.728776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:57.728956 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:57.729001 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.729003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.729708 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.729709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.730247 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:57.730249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.730598 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.730599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:57.730647 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.731269 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:57.731275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:57.733379 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":117,"threads_started":1}
W20260925 15:25:57.733547 10873 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:57.728458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:52153 (local address 127.1.163.126:36453)
0925 15:25:57.728525 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:57.728527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:57.728637 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:25:57.728671 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:57.728672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:57.728687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:57.728758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:57.729077 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.729585 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.730332 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:57.730521 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:57.730584 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:57.733477 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:52153: BlockingRecv error: recv got EOF from 127.1.163.67:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:58.482153 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:58.472957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:35533)
0925 15:25:58.473114 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:58.473127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:58.473188 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:58.473382 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:58.473383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:58.473549 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:58.473604 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.473605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.474308 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.474308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.474870 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.474872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.475227 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.475228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.475270 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.480088 (+ 4818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:58.480096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:58.482080 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
W20260925 15:25:58.482326 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:58.473090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:35533 (local address 127.1.163.126:36453)
0925 15:25:58.473234 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:58.473235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:58.473244 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:25:58.473272 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:58.473272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:58.473283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:58.473362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:58.473703 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.474225 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.474957 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.475156 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.475215 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.482248 (+ 7033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:35533: BlockingRecv error: recv got EOF from 127.1.163.65:35533 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:58.599650 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:58.594680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:49873)
0925 15:25:58.594873 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:58.594884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:58.594951 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:58.595105 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:58.595106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:58.595277 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:58.595327 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.595328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.595974 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.595974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.596515 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.596517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.596952 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.596953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.596998 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.597566 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:58.597571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:58.599573 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260925 15:25:58.599941 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:58.594743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:49873 (local address 127.1.163.126:36453)
0925 15:25:58.594796 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:58.594799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:58.594956 (+ 157us) server_negotiation.cc:407] Connection header received
0925 15:25:58.595019 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:58.595020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:58.595032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:58.595084 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:58.595400 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.595902 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.596663 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.596873 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.596941 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.599890 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:49873: BlockingRecv error: recv got EOF from 127.1.163.66:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:58 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.67@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:58.738951 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:58.734005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.67:53989)
0925 15:25:58.734200 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:58.734214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:58.734310 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:58.734497 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:58.734498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:58.734688 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:58.734736 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.734738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.735432 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.735434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.735981 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:58.735983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.736364 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.736365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:58.736431 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.736959 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:58.736964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:58.738890 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260925 15:25:58.739135 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:58.734116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53989 (local address 127.1.163.126:36453)
0925 15:25:58.734204 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:58.734206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:58.734291 (+ 85us) server_negotiation.cc:407] Connection header received
0925 15:25:58.734402 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:58.734403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:58.734418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:58.734488 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:58.734825 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.735327 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.736078 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:58.736273 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:58.736366 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:58.738993 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.67:53989: BlockingRecv error: recv got EOF from 127.1.163.67:53989 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:25:59 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:59 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.65@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:59.487974 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:59.482780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.65:53365)
0925 15:25:59.482975 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:59.482987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:59.483048 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:59.483242 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:59.483243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:59.483422 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:59.483474 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.483476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.484204 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.484205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.484920 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.484923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.485311 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.485312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.485369 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.485875 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:59.485880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:59.487914 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":106,"threads_started":1}
W20260925 15:25:59.488102 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:59.482901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:53365 (local address 127.1.163.126:36453)
0925 15:25:59.483030 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:59.483032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:59.483045 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:25:59.483103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:59.483104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:59.483115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:59.483184 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:59.483586 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.484108 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.485009 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.485223 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.485312 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.488015 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.65:53365: BlockingRecv error: recv got EOF from 127.1.163.65:53365 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 25 15:25:59 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:25:59 dist-test-slave-hvv5 krb5kdc[9823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.163.66@KRBTEST.COM for oryx/127.1.163.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:25:59.605849 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:59.600404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36453 (local address 127.1.163.66:46783)
0925 15:25:59.600635 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:59.600670 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:59.600776 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:59.600938 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:59.600939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:59.601227 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:59.601299 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.601301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.602007 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.602008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.602873 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.602876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.603249 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.603250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.603294 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.603790 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:25:59.603795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:59.605749 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.163.126:36453: Server oryx/127.1.163.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
W20260925 15:25:59.605916 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:25:59.600562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:46783 (local address 127.1.163.126:36453)
0925 15:25:59.600638 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:59.600640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:59.600745 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:25:59.600854 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:59.600855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:59.600866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:59.600911 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:59.601392 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.601921 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.602987 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.603177 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.603234 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.605853 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.163.66:46783: BlockingRecv error: recv got EOF from 127.1.163.66:46783 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:25:59.651021 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10406
I20260925 15:25:59.655709 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10558
I20260925 15:25:59.659869 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10690
I20260925 15:25:59.664198 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10336
2026-09-25T15:25:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17574 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:25:59Z Disabled control of system clock
I20260925 15:25:59.692365 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37885
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:25:59.775832 10886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:59.776049 10886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:59.776070 10886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:59.777376 10886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:25:59.777421 10886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:59.777433 10886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:59.777446 10886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:25:59.777484 10886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:25:59.778832 10886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:59.779098 10886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:59.779426 10886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:59.784417 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:59.784417 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:59.784835 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:59.784852 10886 server_base.cc:1034] running on GCE node
I20260925 15:25:59.785156 10886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:59.785439 10886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:59.786610 10886 hybrid_clock.cc:648] HybridClock initialized: now 1790349959786600 us; error 30 us; skew 500 ppm
I20260925 15:25:59.787858 10886 webserver.cc:460] Webserver started at http://127.1.163.126:46637/ using document root <none> and password file <none>
I20260925 15:25:59.788100 10886 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:59.788169 10886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:59.788326 10886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:59.789469 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "35db4db3d60d4a50b80dfb4862ffdcb7"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "01e39fbc48006910be8e1f4a2c33b3c7"
server_key_iv: "d0070a52043f55ab540b9b8bd48d09f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.789842 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "35db4db3d60d4a50b80dfb4862ffdcb7"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "01e39fbc48006910be8e1f4a2c33b3c7"
server_key_iv: "d0070a52043f55ab540b9b8bd48d09f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.791121 10886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:25:59.791992 10900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:59.792289 10886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:59.792398 10886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "35db4db3d60d4a50b80dfb4862ffdcb7"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "01e39fbc48006910be8e1f4a2c33b3c7"
server_key_iv: "d0070a52043f55ab540b9b8bd48d09f5"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.792475 10886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:59.802186 10886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:59.802512 10886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:59.802671 10886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:59.806754 10886 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37885
I20260925 15:25:59.806921 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37885 every 8 connection(s)
I20260925 15:25:59.807212 10886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:25:59.807905 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:25:59.810683 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7: Bootstrap starting.
I20260925 15:25:59.811296 10953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7: Neither blocks nor log segments found. Creating new log.
I20260925 15:25:59.811599 10953 log.cc:824] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:25:59.812312 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7: No bootstrap required, opened a new log
I20260925 15:25:59.813987 10953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } }
I20260925 15:25:59.814128 10953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:25:59.814150 10953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35db4db3d60d4a50b80dfb4862ffdcb7, State: Initialized, Role: FOLLOWER
I20260925 15:25:59.814307 10953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } }
I20260925 15:25:59.814389 10953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:25:59.814414 10953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:25:59.814483 10953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:25:59.815037 10953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } }
I20260925 15:25:59.815150 10953 leader_election.cc:302] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35db4db3d60d4a50b80dfb4862ffdcb7; no voters:
I20260925 15:25:59.815339 10953 leader_election.cc:288] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:25:59.815418 10956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:25:59.815618 10956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [term 1 LEADER]: Becoming Leader. State: Replica: 35db4db3d60d4a50b80dfb4862ffdcb7, State: Running, Role: LEADER
I20260925 15:25:59.815626 10953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:25:59.815873 10956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } }
I20260925 15:25:59.816286 10958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35db4db3d60d4a50b80dfb4862ffdcb7. Latest consensus state: current_term: 1 leader_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } } }
I20260925 15:25:59.816373 10958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:59.816438 10957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db4db3d60d4a50b80dfb4862ffdcb7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37885 } } }
I20260925 15:25:59.816504 10957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:25:59.816650 10965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:25:59.816879 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10886
I20260925 15:25:59.816983 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:25:59.817167 10965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:25:59.817188 1677 external_mini_cluster.cc:1428] Setting key 2bc9b596622a433a94a43560061999ed
I20260925 15:25:59.819273 10965 catalog_manager.cc:1347] Generated new cluster ID: 8b64ba48fcd34816aa55ef3551991963
I20260925 15:25:59.819330 10965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:25:59.822949 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:59.817848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.1.163.126:37885)
0925 15:25:59.818046 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:59.818056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:59.818325 (+ 269us) server_negotiation.cc:407] Connection header received
0925 15:25:59.818449 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:59.818458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:59.818533 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:59.818694 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:59.818980 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.820059 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.820957 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.821400 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.821532 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.822241 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:59.822244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:59.822245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:59.822429 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:59.822461 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:59.822472 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:59.822550 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:59.822639 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:25:59.822721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260925 15:25:59.829442 10965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:25:59.829996 10965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:25:59.833364 10965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35db4db3d60d4a50b80dfb4862ffdcb7: Generated new TSK 0
I20260925 15:25:59.833582 10965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:25:59.835938 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--builtin_ntp_servers=127.1.163.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:25:59.925253 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:25:59.925500 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:25:59.925539 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:25:59.926806 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:25:59.926878 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:25:59.926926 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:25:59.928376 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:25:59.928681 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:25:59.928999 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:25:59.934458 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:25:59.934571 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:59.934644 10977 server_base.cc:1034] running on GCE node
W20260925 15:25:59.934453 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:25:59.934983 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:25:59.935266 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:25:59.936441 10977 hybrid_clock.cc:648] HybridClock initialized: now 1790349959936426 us; error 32 us; skew 500 ppm
I20260925 15:25:59.937837 10977 webserver.cc:460] Webserver started at http://127.1.163.65:37421/ using document root <none> and password file <none>
I20260925 15:25:59.938109 10977 fs_manager.cc:359] Metadata directory not provided
I20260925 15:25:59.938184 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:25:59.938341 10977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:25:59.939450 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "5c87b9d68274466aa94f542e0a6c0b19"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "8b11c111d65dbf63cc23e594f19e116f"
server_key_iv: "0034dd1aa0c47552ddb7613bbeafcc78"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.939831 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "5c87b9d68274466aa94f542e0a6c0b19"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "8b11c111d65dbf63cc23e594f19e116f"
server_key_iv: "0034dd1aa0c47552ddb7613bbeafcc78"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.941341 10977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:25:59.942246 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:59.942477 10977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:25:59.942605 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "5c87b9d68274466aa94f542e0a6c0b19"
format_stamp: "Formatted at 2026-09-25 15:25:59 on dist-test-slave-hvv5"
server_key: "8b11c111d65dbf63cc23e594f19e116f"
server_key_iv: "0034dd1aa0c47552ddb7613bbeafcc78"
server_key_version: "encryptionkey@0"
I20260925 15:25:59.942725 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:25:59.969046 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:25:59.969401 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:25:59.969573 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:25:59.969834 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:25:59.970175 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:25:59.970232 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:59.970270 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:25:59.970320 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:25:59.976413 10977 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35841
I20260925 15:25:59.976468 11104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35841 every 8 connection(s)
I20260925 15:25:59.976847 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:25:59.982738 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:59.977683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42135 (local address 127.1.163.126:37885)
0925 15:25:59.977761 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:25:59.977763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:25:59.977980 (+ 217us) server_negotiation.cc:407] Connection header received
0925 15:25:59.978462 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:25:59.978463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:25:59.978475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:25:59.978524 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:25:59.979241 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.979740 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.981236 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.981432 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.981500 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.982373 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:25:59.982375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:25:59.982375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:25:59.982383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:25:59.982396 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:25:59.982400 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:25:59.982452 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:25:59.982653 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:25:59.982678 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:25:59.982795 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:25:59.977633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37885 (local address 127.1.163.65:42135)
0925 15:25:59.977851 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:25:59.977888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:25:59.978377 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:25:59.978583 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:25:59.978587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:25:59.978595 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:25:59.979150 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.979156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.979854 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.979855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.981106 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:25:59.981109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:25:59.981527 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:25:59.981528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:25:59.981644 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:25:59.982204 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:25:59.982204 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:25:59.982234 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:25:59.982257 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:25:59.982469 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:25:59.982473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:25:59.982476 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:25:59.982536 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:25:59.982623 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260925 15:25:59.983390 11105 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37885
I20260925 15:25:59.983486 11105 heartbeater.cc:464] Registering TS with master...
I20260925 15:25:59.983639 11105 heartbeater.cc:511] Master 127.1.163.126:37885 requested a full tablet report, sending...
I20260925 15:25:59.984095 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 10977
I20260925 15:25:59.984194 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:25:59.984177 10917 ts_manager.cc:194] Registered new tserver with Master: 5c87b9d68274466aa94f542e0a6c0b19 (127.1.163.65:35841)
I20260925 15:25:59.984403 1677 external_mini_cluster.cc:1428] Setting key a13beb3bfc779549e609cfbedbb43b45
I20260925 15:25:59.985006 10917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.65:42135
I20260925 15:25:59.985522 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--builtin_ntp_servers=127.1.163.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:26:00.067080 11108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.067304 11108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.067327 11108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.068588 11108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.068632 11108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.068658 11108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:26:00.070044 11108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.070324 11108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.070650 11108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:00.075817 11114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.075891 11113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.075817 11116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.076130 11108 server_base.cc:1034] running on GCE node
I20260925 15:26:00.076362 11108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.076622 11108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.077761 11108 hybrid_clock.cc:648] HybridClock initialized: now 1790349960077736 us; error 44 us; skew 500 ppm
I20260925 15:26:00.079175 11108 webserver.cc:460] Webserver started at http://127.1.163.66:41975/ using document root <none> and password file <none>
I20260925 15:26:00.079447 11108 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.079496 11108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.079670 11108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.080852 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "ef7e546190284a4291578b660b4ec9f5"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "0acfcfc9f0b0a2df6adf86855b1ebcd0"
server_key_iv: "2b983bcbb8006a85a959b6e10cf41e14"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.081228 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "ef7e546190284a4291578b660b4ec9f5"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "0acfcfc9f0b0a2df6adf86855b1ebcd0"
server_key_iv: "2b983bcbb8006a85a959b6e10cf41e14"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.082579 11108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:26:00.083310 11122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.083514 11108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:26:00.083622 11108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "ef7e546190284a4291578b660b4ec9f5"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "0acfcfc9f0b0a2df6adf86855b1ebcd0"
server_key_iv: "2b983bcbb8006a85a959b6e10cf41e14"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.083693 11108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.116994 11108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.117292 11108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.117413 11108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.117604 11108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:00.117956 11108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:00.117991 11108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.118048 11108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:00.118080 11108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.124851 11108 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:46261
I20260925 15:26:00.124894 11235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:46261 every 8 connection(s)
I20260925 15:26:00.125289 11108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:26:00.130924 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11108
I20260925 15:26:00.131068 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:26:00.131157 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.126151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40993 (local address 127.1.163.126:37885)
0925 15:26:00.126231 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.126233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.126492 (+ 259us) server_negotiation.cc:407] Connection header received
0925 15:26:00.127033 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.127034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.127045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.127106 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.127784 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.128320 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.129490 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.129683 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.129756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:00.130816 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.130817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.130818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.130824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.130844 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.130851 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.130916 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.131017 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.131056 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:26:00.131282 1677 external_mini_cluster.cc:1428] Setting key 20e5e5e3da9a88f540f5acaf713496fa
I20260925 15:26:00.131400 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.126149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37885 (local address 127.1.163.66:40993)
0925 15:26:00.126360 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:00.126396 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:00.126910 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:00.127174 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:00.127177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:00.127186 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:00.127684 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:00.127688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.128414 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.128415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:00.129386 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:00.129389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.129811 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.129812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:00.129936 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:00.130621 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:26:00.130621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:00.130690 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:26:00.130717 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:00.130933 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:00.130938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:00.130940 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:26:00.130993 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:26:00.131133 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260925 15:26:00.131855 11236 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37885
I20260925 15:26:00.131965 11236 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:00.132148 11236 heartbeater.cc:511] Master 127.1.163.126:37885 requested a full tablet report, sending...
I20260925 15:26:00.132368 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--builtin_ntp_servers=127.1.163.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:26:00.132629 10916 ts_manager.cc:194] Registered new tserver with Master: ef7e546190284a4291578b660b4ec9f5 (127.1.163.66:46261)
I20260925 15:26:00.133031 10916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.66:40993
W20260925 15:26:00.217535 11239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.217748 11239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.217770 11239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.218896 11239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.218930 11239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.218954 11239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:26:00.220228 11239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.220425 11239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.220753 11239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:00.225677 11247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.225683 11245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.225886 11239 server_base.cc:1034] running on GCE node
W20260925 15:26:00.225919 11244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.226164 11239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.226424 11239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.227591 11239 hybrid_clock.cc:648] HybridClock initialized: now 1790349960227577 us; error 42 us; skew 500 ppm
I20260925 15:26:00.228889 11239 webserver.cc:460] Webserver started at http://127.1.163.67:46741/ using document root <none> and password file <none>
I20260925 15:26:00.229142 11239 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.229218 11239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.229375 11239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.230513 11239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "ba5357c69ae946cfa16d2415b4211c5e"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "a09e9e43a2079704f563089b2c7a6e4f"
server_key_iv: "9faa61d73b016fe670c35e4b38d490f6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.230878 11239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "ba5357c69ae946cfa16d2415b4211c5e"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "a09e9e43a2079704f563089b2c7a6e4f"
server_key_iv: "9faa61d73b016fe670c35e4b38d490f6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.232215 11239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:26:00.233119 11253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.233289 11239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:26:00.233404 11239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "ba5357c69ae946cfa16d2415b4211c5e"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "a09e9e43a2079704f563089b2c7a6e4f"
server_key_iv: "9faa61d73b016fe670c35e4b38d490f6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.233481 11239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.255182 11239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.255525 11239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.255681 11239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.255930 11239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:00.256273 11239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:00.256330 11239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.256367 11239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:00.256397 11239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.262149 11239 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:33495
I20260925 15:26:00.262197 11366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:33495 every 8 connection(s)
I20260925 15:26:00.262550 11239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:26:00.267814 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.263286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53007 (local address 127.1.163.126:37885)
0925 15:26:00.263364 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.263366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.263576 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:26:00.264059 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.264060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.264069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.264134 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.264895 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.265420 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.266392 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.266603 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.266709 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:00.267541 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.267542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.267543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.267547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.267563 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.267567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.267612 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.267726 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.267755 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:26:00.267822 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11239
I20260925 15:26:00.268042 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:26:00.268106 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.263239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37885 (local address 127.1.163.67:53007)
0925 15:26:00.263438 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:00.263482 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:00.263973 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:00.264212 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:00.264216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:00.264224 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:00.264783 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:00.264787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.265509 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.265510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:00.266286 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:00.266288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:00.266730 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:00.266731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:00.266845 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:00.267398 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:26:00.267398 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:00.267429 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:26:00.267451 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:00.267641 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:00.267646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:00.267650 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:26:00.267724 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:26:00.267843 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260925 15:26:00.268266 1677 external_mini_cluster.cc:1428] Setting key 8ab4b469882dbd2edf4922b106504465
I20260925 15:26:00.268603 11367 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37885
I20260925 15:26:00.268699 11367 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:00.268893 11367 heartbeater.cc:511] Master 127.1.163.126:37885 requested a full tablet report, sending...
I20260925 15:26:00.269225 10916 ts_manager.cc:194] Registered new tserver with Master: ba5357c69ae946cfa16d2415b4211c5e (127.1.163.67:33495)
I20260925 15:26:00.269563 10916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.67:53007
I20260925 15:26:00.280135 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:26:00.282207 10975 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:26:00.281604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.1.163.126:37885)
0925 15:26:00.281690 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.281693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.281794 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:26:00.281886 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.281887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.281901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.281977 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.282124 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45776: BlockingRecv error: recv got EOF from 127.0.0.1:45776 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:26:00.282253 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10977
I20260925 15:26:00.287083 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11108
I20260925 15:26:00.291404 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11239
I20260925 15:26:00.295848 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 10886
2026-09-25T15:26:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (630 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:26:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:26:00Z Disabled control of system clock
I20260925 15:26:00.322698 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36559
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:26:00.405015 11378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.405238 11378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.405260 11378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.406476 11378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:26:00.406549 11378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.406574 11378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.406594 11378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:26:00.406613 11378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:26:00.407996 11378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36559
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.408208 11378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.408468 11378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:26:00.413628 11378 server_base.cc:1034] running on GCE node
W20260925 15:26:00.413612 11384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.413610 11383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.413600 11386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.414086 11378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.414409 11378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.415589 11378 hybrid_clock.cc:648] HybridClock initialized: now 1790349960415565 us; error 40 us; skew 500 ppm
I20260925 15:26:00.417049 11378 webserver.cc:460] Webserver started at http://127.1.163.126:35401/ using document root <none> and password file <none>
I20260925 15:26:00.417323 11378 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.417397 11378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.417549 11378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.418645 11378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "a55a4710f5114aa09d7ee9b7d9fccbed"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "3baf7fb7ff995aef9d34bbf3ab9018c7"
server_key_iv: "06661d9657bb8c26557566934d169c5b"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.419020 11378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "a55a4710f5114aa09d7ee9b7d9fccbed"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "3baf7fb7ff995aef9d34bbf3ab9018c7"
server_key_iv: "06661d9657bb8c26557566934d169c5b"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.420344 11378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:26:00.421207 11392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.421386 11378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:26:00.421490 11378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "a55a4710f5114aa09d7ee9b7d9fccbed"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "3baf7fb7ff995aef9d34bbf3ab9018c7"
server_key_iv: "06661d9657bb8c26557566934d169c5b"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.421577 11378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.425071 11378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.425360 11378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.425508 11378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.429513 11378 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36559
I20260925 15:26:00.429544 11444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36559 every 8 connection(s)
I20260925 15:26:00.429898 11378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:26:00.430558 11445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:00.433483 11445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed: Bootstrap starting.
I20260925 15:26:00.434098 11445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:00.434409 11445 log.cc:824] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:00.435138 11445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed: No bootstrap required, opened a new log
I20260925 15:26:00.436852 11445 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } }
I20260925 15:26:00.437011 11445 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:00.437034 11445 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a4710f5114aa09d7ee9b7d9fccbed, State: Initialized, Role: FOLLOWER
I20260925 15:26:00.437161 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11378
I20260925 15:26:00.437266 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:26:00.437214 11445 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } }
I20260925 15:26:00.437315 11445 raft_consensus.cc:388] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:26:00.437388 11445 raft_consensus.cc:482] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:26:00.437444 11445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:00.437455 1677 external_mini_cluster.cc:1428] Setting key 1185559dd5b370c5b71e91d981ba32ed
I20260925 15:26:00.438159 11445 raft_consensus.cc:504] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } }
I20260925 15:26:00.438277 11445 leader_election.cc:302] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a55a4710f5114aa09d7ee9b7d9fccbed; no voters:
I20260925 15:26:00.438442 11445 leader_election.cc:288] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:26:00.438499 11450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:26:00.438644 11450 raft_consensus.cc:686] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [term 1 LEADER]: Becoming Leader. State: Replica: a55a4710f5114aa09d7ee9b7d9fccbed, State: Running, Role: LEADER
I20260925 15:26:00.438714 11445 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:26:00.438736 11450 consensus_queue.cc:237] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } }
I20260925 15:26:00.439112 11450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a55a4710f5114aa09d7ee9b7d9fccbed. Latest consensus state: current_term: 1 leader_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } } }
I20260925 15:26:00.439198 11450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [sys.catalog]: This master's current role is: LEADER
I20260925 15:26:00.439462 11458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:26:00.439850 11452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a55a4710f5114aa09d7ee9b7d9fccbed" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36559 } } }
I20260925 15:26:00.440021 11458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:26:00.440021 11452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed [sys.catalog]: This master's current role is: LEADER
I20260925 15:26:00.440330 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.438005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.1.163.126:36559)
0925 15:26:00.438220 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.438229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.438253 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:26:00.438408 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.438414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.438479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.438637 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.439773 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.439777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.439778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.439783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.439833 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.439847 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.439904 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.440074 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.440134 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 15:26:00.445751 11458 catalog_manager.cc:1347] Generated new cluster ID: cebd2332ee19448bb533f9f1662d51a5
I20260925 15:26:00.445824 11458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:26:00.471416 11458 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:26:00.471634 11458 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:26:00.473830 11458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a55a4710f5114aa09d7ee9b7d9fccbed: Generated new TSK 0
I20260925 15:26:00.474045 11458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:26:00.478521 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--builtin_ntp_servers=127.1.163.84:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:26:00.562956 11469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.563210 11469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.563251 11469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.564495 11469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.564584 11469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.564634 11469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:26:00.566136 11469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.566449 11469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.566774 11469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:00.571995 11475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.572052 11474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.572280 11477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.572557 11469 server_base.cc:1034] running on GCE node
I20260925 15:26:00.572778 11469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.573056 11469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.574235 11469 hybrid_clock.cc:648] HybridClock initialized: now 1790349960574225 us; error 47 us; skew 500 ppm
I20260925 15:26:00.575588 11469 webserver.cc:460] Webserver started at http://127.1.163.65:33707/ using document root <none> and password file <none>
I20260925 15:26:00.575872 11469 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.575947 11469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.576102 11469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.577263 11469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "830aad5978fb43779d7f15f2cee05633"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "5a1496a3e17c21e35cf00abe375a17a8"
server_key_iv: "d9aaf780c018d1305c1e48ba21807a39"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.577630 11469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "830aad5978fb43779d7f15f2cee05633"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "5a1496a3e17c21e35cf00abe375a17a8"
server_key_iv: "d9aaf780c018d1305c1e48ba21807a39"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.578999 11469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:26:00.579774 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.579950 11469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:26:00.580054 11469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "830aad5978fb43779d7f15f2cee05633"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "5a1496a3e17c21e35cf00abe375a17a8"
server_key_iv: "d9aaf780c018d1305c1e48ba21807a39"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.580129 11469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.583606 11469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.583891 11469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.584040 11469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.584264 11469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:00.584633 11469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:00.584690 11469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.584733 11469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:00.584758 11469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.590643 11469 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:44279
I20260925 15:26:00.590685 11596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:44279 every 8 connection(s)
I20260925 15:26:00.591028 11469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:26:00.593657 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11469
I20260925 15:26:00.593729 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:26:00.593930 1677 external_mini_cluster.cc:1428] Setting key 703ebc89cb560bc976da20941d703d82
I20260925 15:26:00.594349 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.591761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:44915 (local address 127.1.163.126:36559)
0925 15:26:00.591838 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.591840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.592090 (+ 250us) server_negotiation.cc:407] Connection header received
0925 15:26:00.592710 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.592711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.592719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.592777 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.594036 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.594037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.594038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.594042 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.594060 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.594065 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.594142 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.594259 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.594281 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:26:00.594532 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.591743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36559 (local address 127.1.163.65:44915)
0925 15:26:00.591959 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:00.591995 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:00.592593 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:00.592854 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:00.592859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:00.592875 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:00.593859 (+ 984us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:26:00.593859 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:00.593888 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:26:00.593919 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:00.594138 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:00.594144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:00.594147 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:26:00.594228 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:26:00.594295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260925 15:26:00.595024 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--builtin_ntp_servers=127.1.163.84:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:26:00.595014 11597 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36559
I20260925 15:26:00.595134 11597 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:00.595279 11597 heartbeater.cc:511] Master 127.1.163.126:36559 requested a full tablet report, sending...
I20260925 15:26:00.595841 11409 ts_manager.cc:194] Registered new tserver with Master: 830aad5978fb43779d7f15f2cee05633 (127.1.163.65:44279)
W20260925 15:26:00.679404 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.679639 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.679665 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.680977 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.681026 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.681056 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:26:00.682538 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.682794 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.683145 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:00.688851 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.688908 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.689007 11608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.689142 11600 server_base.cc:1034] running on GCE node
I20260925 15:26:00.689426 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.689714 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.690918 11600 hybrid_clock.cc:648] HybridClock initialized: now 1790349960690886 us; error 42 us; skew 500 ppm
I20260925 15:26:00.692266 11600 webserver.cc:460] Webserver started at http://127.1.163.66:33121/ using document root <none> and password file <none>
I20260925 15:26:00.692589 11600 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.692672 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.692821 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.693977 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "7a8aebf648d044b29f202e7f276c5ee3"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "92df51f99a329081dc9397e9830cb72f"
server_key_iv: "bf2654882e5db36152ecf5661be94a2e"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.694361 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "7a8aebf648d044b29f202e7f276c5ee3"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "92df51f99a329081dc9397e9830cb72f"
server_key_iv: "bf2654882e5db36152ecf5661be94a2e"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.695696 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:26:00.696436 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.696635 11600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:26:00.696738 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "7a8aebf648d044b29f202e7f276c5ee3"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "92df51f99a329081dc9397e9830cb72f"
server_key_iv: "bf2654882e5db36152ecf5661be94a2e"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.696861 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.700384 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.700721 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.700887 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.701128 11600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:00.701471 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:00.701529 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.701570 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:00.701596 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.707465 11600 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:44851
I20260925 15:26:00.707551 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:44851 every 8 connection(s)
I20260925 15:26:00.707904 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:26:00.710436 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11600
I20260925 15:26:00.710554 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:26:00.710809 1677 external_mini_cluster.cc:1428] Setting key b8f57bd3b018baabf6b9bdc3a9269d05
I20260925 15:26:00.711411 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.708736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:39503 (local address 127.1.163.126:36559)
0925 15:26:00.708812 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.708814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.709054 (+ 240us) server_negotiation.cc:407] Connection header received
0925 15:26:00.709599 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.709601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.709607 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.709666 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.711122 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.711124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.711124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.711129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.711147 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.711153 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.711215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.711325 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.711340 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:26:00.711637 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.708750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36559 (local address 127.1.163.66:39503)
0925 15:26:00.708930 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:00.708965 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:00.709482 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:00.709751 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:00.709755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:00.709763 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:00.710913 (+ 1150us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:26:00.710914 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:00.710956 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:26:00.710999 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:00.711236 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:00.711242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:00.711245 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:26:00.711318 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:26:00.711376 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260925 15:26:00.711958 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--builtin_ntp_servers=127.1.163.84:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:26:00.712229 11728 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36559
I20260925 15:26:00.712322 11728 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:00.712738 11728 heartbeater.cc:511] Master 127.1.163.126:36559 requested a full tablet report, sending...
I20260925 15:26:00.713201 11409 ts_manager.cc:194] Registered new tserver with Master: 7a8aebf648d044b29f202e7f276c5ee3 (127.1.163.66:44851)
W20260925 15:26:00.796337 11731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:00.796630 11731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:00.796659 11731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:00.797914 11731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:00.797959 11731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:00.797988 11731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:26:00.799412 11731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:00.799750 11731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:00.800071 11731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:00.805131 11739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.805151 11736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:00.805218 11737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:00.805269 11731 server_base.cc:1034] running on GCE node
I20260925 15:26:00.805557 11731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:00.805832 11731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:00.806986 11731 hybrid_clock.cc:648] HybridClock initialized: now 1790349960806976 us; error 35 us; skew 500 ppm
I20260925 15:26:00.808359 11731 webserver.cc:460] Webserver started at http://127.1.163.67:38399/ using document root <none> and password file <none>
I20260925 15:26:00.808688 11731 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:00.808764 11731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:00.808938 11731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:00.810141 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "6aff1d190bc8485a9cb41e55e75ea3cc"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "bd2d745d1e9ca14ac793ea9cb892a9da"
server_key_iv: "60d20fd392285f36a8481e4f3c3f5ea6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.810595 11731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "6aff1d190bc8485a9cb41e55e75ea3cc"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "bd2d745d1e9ca14ac793ea9cb892a9da"
server_key_iv: "60d20fd392285f36a8481e4f3c3f5ea6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.811937 11731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:26:00.812799 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.812969 11731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:26:00.813071 11731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "6aff1d190bc8485a9cb41e55e75ea3cc"
format_stamp: "Formatted at 2026-09-25 15:26:00 on dist-test-slave-hvv5"
server_key: "bd2d745d1e9ca14ac793ea9cb892a9da"
server_key_iv: "60d20fd392285f36a8481e4f3c3f5ea6"
server_key_version: "encryptionkey@0"
I20260925 15:26:00.813158 11731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:00.816679 11731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:00.816975 11731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:00.817155 11731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:00.817387 11731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:00.817715 11731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:00.817770 11731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.817812 11731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:00.817838 11731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:00.824344 11731 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:36969
I20260925 15:26:00.824775 11858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:36969 every 8 connection(s)
I20260925 15:26:00.824870 11731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:26:00.827425 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11731
I20260925 15:26:00.827541 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:26:00.827752 1677 external_mini_cluster.cc:1428] Setting key 97075e7734b68b60edb9c0b692b883f0
I20260925 15:26:00.828751 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.825851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:60119 (local address 127.1.163.126:36559)
0925 15:26:00.825938 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.825940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.826172 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:26:00.826658 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.826659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.826665 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.826728 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.828297 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:00.828298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:00.828298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:26:00.828302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:00.828343 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:26:00.828348 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:00.828402 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:00.828623 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:26:00.828640 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:26:00.828969 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:00.825785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36559 (local address 127.1.163.67:60119)
0925 15:26:00.826031 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:00.826067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:00.826569 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:00.826784 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:00.826788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:00.826797 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:00.828010 (+ 1213us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:26:00.828010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:00.828055 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:26:00.828096 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:00.828435 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:00.828444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:00.828448 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:26:00.828633 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 15:26:00.828715 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":132,"threads_started":1}
I20260925 15:26:00.829591 11859 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36559
I20260925 15:26:00.829689 11859 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:00.829893 11859 heartbeater.cc:511] Master 127.1.163.126:36559 requested a full tablet report, sending...
I20260925 15:26:00.830484 11409 ts_manager.cc:194] Registered new tserver with Master: 6aff1d190bc8485a9cb41e55e75ea3cc (127.1.163.67:36969)
I20260925 15:26:00.840219 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:26:00.842362 11449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:26:00.841761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.1.163.126:36559)
0925 15:26:00.841821 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:00.841823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:00.842001 (+ 178us) server_negotiation.cc:407] Connection header received
0925 15:26:00.842034 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:00.842035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:00.842041 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:00.842090 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:00.842277 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44832: BlockingRecv error: recv got EOF from 127.0.0.1:44832 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:26:00.842437 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11469
I20260925 15:26:00.847023 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11600
I20260925 15:26:00.851933 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11731
I20260925 15:26:00.856349 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11378
2026-09-25T15:26:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (561 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:26:00 dist-test-slave-hvv5 krb5kdc[11869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:26:00 dist-test-slave-hvv5 krb5kdc[11869](info): set up 2 sockets
Sep 25 15:26:00 dist-test-slave-hvv5 krb5kdc[11869](info): commencing operation
krb5kdc: starting...
W20260925 15:26:01.910589 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:26:01 dist-test-slave-hvv5 krb5kdc[11869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:26:02.037124 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:39837
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:39837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:02.119061 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:02.119346 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:02.119386 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:02.120612 11882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:26:02.120687 11882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:02.120711 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:02.120819 11882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:26:02.120857 11882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:26:02.122447 11882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:39837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:39837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:02.122752 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:02.122992 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:02.128129 11887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.128103 11890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.128114 11888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.128695 11882 server_base.cc:1034] running on GCE node
I20260925 15:26:02.128939 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:02.129248 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:02.130422 11882 hybrid_clock.cc:648] HybridClock initialized: now 1790349962130410 us; error 47 us; skew 500 ppm
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:02.132965 11882 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:26:02.133477 11882 webserver.cc:460] Webserver started at http://127.1.163.126:37665/ using document root <none> and password file <none>
I20260925 15:26:02.133719 11882 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:02.133792 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:02.133937 11882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:02.135011 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "302640e22a0f4c1980e6281a9809f7fd"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "68751430ffcd378449e3c6aec9aaec4a"
server_key_iv: "1f7bed0778d8e23de21c61f7fe62c805"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.135454 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "302640e22a0f4c1980e6281a9809f7fd"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "68751430ffcd378449e3c6aec9aaec4a"
server_key_iv: "1f7bed0778d8e23de21c61f7fe62c805"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.136951 11882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:26:02.137704 11897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.137928 11882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:26:02.138027 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "302640e22a0f4c1980e6281a9809f7fd"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "68751430ffcd378449e3c6aec9aaec4a"
server_key_iv: "1f7bed0778d8e23de21c61f7fe62c805"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.138110 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:02.149564 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:02.152137 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:02.152314 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:02.158066 11882 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:39837
I20260925 15:26:02.158149 11949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:39837 every 8 connection(s)
I20260925 15:26:02.158588 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:26:02.160310 11950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:02.163290 11950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: Bootstrap starting.
I20260925 15:26:02.163975 11950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:02.164263 11950 log.cc:824] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:02.165128 11950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: No bootstrap required, opened a new log
I20260925 15:26:02.166491 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11882
I20260925 15:26:02.166591 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:26:02.166767 1677 external_mini_cluster.cc:1428] Setting key 425f3e1ad5e71dae63c9ec84e380c660
I20260925 15:26:02.167688 11950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } }
I20260925 15:26:02.167855 11950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:02.167896 11950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302640e22a0f4c1980e6281a9809f7fd, State: Initialized, Role: FOLLOWER
I20260925 15:26:02.168025 11950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } }
I20260925 15:26:02.168110 11950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:26:02.168152 11950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:26:02.168201 11950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:02.169384 11950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } }
I20260925 15:26:02.169505 11950 leader_election.cc:302] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302640e22a0f4c1980e6281a9809f7fd; no voters:
I20260925 15:26:02.169667 11950 leader_election.cc:288] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:26:02.169739 11955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:26:02.169898 11955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [term 1 LEADER]: Becoming Leader. State: Replica: 302640e22a0f4c1980e6281a9809f7fd, State: Running, Role: LEADER
I20260925 15:26:02.169938 11950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:26:02.169986 11955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } }
I20260925 15:26:02.170306 11956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302640e22a0f4c1980e6281a9809f7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } } }
I20260925 15:26:02.170404 11956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [sys.catalog]: This master's current role is: LEADER
I20260925 15:26:02.170557 11957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302640e22a0f4c1980e6281a9809f7fd. Latest consensus state: current_term: 1 leader_uuid: "302640e22a0f4c1980e6281a9809f7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302640e22a0f4c1980e6281a9809f7fd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 39837 } } }
I20260925 15:26:02.170625 11957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd [sys.catalog]: This master's current role is: LEADER
I20260925 15:26:02.170753 11961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:26:02.171263 11961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:26:02.172927 11961 catalog_manager.cc:1347] Generated new cluster ID: 2c73e3a325314c27a40e22e7fbf677e5
I20260925 15:26:02.172982 11961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:02.178757 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.167469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.1.163.126:39837)
0925 15:26:02.168303 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.168314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.168348 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:26:02.168502 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.168509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.168599 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.168749 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:02.169575 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.171071 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.172018 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.172618 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.172719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.175251 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:02.175256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:02.175256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:02.175268 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:02.177133 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.177531 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.177534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.177535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.177654 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.177874 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.177874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.177875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.178132 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:02.178296 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.178409 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.178559 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":90,"threads_started":1}
I20260925 15:26:02.183655 11961 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:26:02.184365 11961 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:26:02.188465 11961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: Generated new TSK 0
I20260925 15:26:02.188687 11961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:26:02.220959 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:02.311031 11976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:02.311267 11976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:02.311309 11976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:02.312640 11976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:02.312705 11976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:02.312754 11976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:26:02.314342 11976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:02.314632 11976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:02.314941 11976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:02.320139 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.320139 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.320163 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.320276 11976 server_base.cc:1034] running on GCE node
I20260925 15:26:02.320617 11976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:02.320894 11976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:02.322039 11976 hybrid_clock.cc:648] HybridClock initialized: now 1790349962322026 us; error 40 us; skew 500 ppm
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:02.324725 11976 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:26:02.325141 11976 webserver.cc:460] Webserver started at http://127.1.163.65:46483/ using document root <none> and password file <none>
I20260925 15:26:02.325350 11976 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:02.325392 11976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:02.325482 11976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:02.326611 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "d7672261426c480bbc0754ed6dc8692b"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "6ba34ffd62e836194b41a7fa0808d8cd"
server_key_iv: "bfaf6c7674ff92c5cd9ae8ae2ed6b53d"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.326927 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "d7672261426c480bbc0754ed6dc8692b"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "6ba34ffd62e836194b41a7fa0808d8cd"
server_key_iv: "bfaf6c7674ff92c5cd9ae8ae2ed6b53d"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.328202 11976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:26:02.329044 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.329255 11976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:26:02.329365 11976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "d7672261426c480bbc0754ed6dc8692b"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "6ba34ffd62e836194b41a7fa0808d8cd"
server_key_iv: "bfaf6c7674ff92c5cd9ae8ae2ed6b53d"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.329437 11976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:02.367714 11976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:02.371203 11976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:02.371431 11976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:02.371726 11976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:02.372280 11976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:02.372399 11976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.372452 11976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:02.372484 11976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.380707 11976 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:45801
I20260925 15:26:02.380836 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:45801 every 8 connection(s)
I20260925 15:26:02.381161 11976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:26:02.387387 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 11976
I20260925 15:26:02.387487 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:26:02.387692 1677 external_mini_cluster.cc:1428] Setting key 418965d748c21c33616b8dd02222f2e7
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:02.395411 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.382060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:47577 (local address 127.1.163.126:39837)
0925 15:26:02.382109 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.382112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.382379 (+ 267us) server_negotiation.cc:407] Connection header received
0925 15:26:02.382944 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.382945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.382964 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.383028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:02.384649 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.385498 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.387256 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.387710 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.387802 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.391287 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:02.391289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:02.391290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:02.391313 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:02.393306 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.393928 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.393929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.393930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.394009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.394358 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.394359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.394359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.394567 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:02.394674 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.395137 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.395281 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:26:02.396327 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.382061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39837 (local address 127.1.163.65:47577)
0925 15:26:02.382246 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.382285 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.382831 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.383167 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.383172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.384107 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:02.384463 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.384471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.385626 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.385628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.387102 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.387105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.387919 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.387920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.388135 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.388919 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:02.388934 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:02.391160 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:02.393401 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.393410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.393413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.393776 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.394097 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.394098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.394099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.394267 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.394662 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:02.394671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:02.394943 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.395757 (+ 814us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.396052 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260925 15:26:02.397295 12105 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39837
I20260925 15:26:02.397405 12105 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:02.397572 12105 heartbeater.cc:511] Master 127.1.163.126:39837 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:26:02.398123 11914 ts_manager.cc:194] Registered new tserver with Master: d7672261426c480bbc0754ed6dc8692b (127.1.163.65:45801)
I20260925 15:26:02.398823 11914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:47577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:26:02.415637 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:02.499083 12110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:02.499338 12110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:02.499382 12110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:02.500605 12110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:02.500671 12110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:02.500717 12110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:26:02.502223 12110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:02.502502 12110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:02.502741 12110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:02.507959 12115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.508028 12118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.508054 12110 server_base.cc:1034] running on GCE node
W20260925 15:26:02.507959 12116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.508373 12110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:02.508677 12110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:02.509841 12110 hybrid_clock.cc:648] HybridClock initialized: now 1790349962509828 us; error 29 us; skew 500 ppm
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:02.512424 12110 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:26:02.512988 12110 webserver.cc:460] Webserver started at http://127.1.163.66:35371/ using document root <none> and password file <none>
I20260925 15:26:02.513239 12110 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:02.513309 12110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:02.513463 12110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:02.514559 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "38a2249cb1934ce2813f94efc2dd0e5c"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "88490cc3b4488a68d4d75bb56464bc74"
server_key_iv: "340663628b8a73ab4e5e42fd9ac3f539"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.514926 12110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "38a2249cb1934ce2813f94efc2dd0e5c"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "88490cc3b4488a68d4d75bb56464bc74"
server_key_iv: "340663628b8a73ab4e5e42fd9ac3f539"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.516260 12110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:26:02.517063 12125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.517241 12110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.517338 12110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "38a2249cb1934ce2813f94efc2dd0e5c"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "88490cc3b4488a68d4d75bb56464bc74"
server_key_iv: "340663628b8a73ab4e5e42fd9ac3f539"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.517419 12110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:02.527618 12110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:02.529906 12110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:02.530078 12110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:02.530361 12110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:02.530735 12110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:02.530773 12110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.530822 12110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:02.530846 12110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.537097 12110 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39231
I20260925 15:26:02.537134 12238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39231 every 8 connection(s)
I20260925 15:26:02.537510 12110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:26:02.540691 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 12110
I20260925 15:26:02.540792 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:26:02.541008 1677 external_mini_cluster.cc:1428] Setting key a26326e99e62a042fefd719f4e4e965e
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:02.548023 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.538326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:42009 (local address 127.1.163.126:39837)
0925 15:26:02.538387 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.538390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.538583 (+ 193us) server_negotiation.cc:407] Connection header received
0925 15:26:02.539080 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.539081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.539091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.539153 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:02.540136 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.540787 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.542224 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.542420 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.542477 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.544810 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:02.544811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:02.544812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:02.544820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:02.546542 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.546987 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.546989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.546989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.547056 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.547303 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.547304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.547305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.547463 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:02.547537 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.547830 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.547920 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:26:02.548092 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.538286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39837 (local address 127.1.163.66:42009)
0925 15:26:02.538452 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.538495 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.538994 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.539250 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.539254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.539838 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:02.540048 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.540052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.540902 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.540903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.542145 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.542148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.542512 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.542513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.542676 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.543220 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:02.543232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:02.544716 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:02.546633 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.546640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.546642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.546909 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.547129 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.547130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.547130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.547233 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.547531 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:02.547535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:02.547649 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.547773 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.547896 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260925 15:26:02.548636 12239 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39837
I20260925 15:26:02.548743 12239 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:02.548954 12239 heartbeater.cc:511] Master 127.1.163.126:39837 requested a full tablet report, sending...
I20260925 15:26:02.549414 11914 ts_manager.cc:194] Registered new tserver with Master: 38a2249cb1934ce2813f94efc2dd0e5c (127.1.163.66:39231)
I20260925 15:26:02.549829 11914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:42009
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:26:02.568079 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:02.651654 12244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:02.651902 12244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:02.651926 12244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:02.653175 12244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:02.653226 12244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:02.653255 12244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:26:02.654882 12244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:39837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:02.655192 12244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:02.655457 12244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:02.661311 12249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:02.661316 12252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.661499 12244 server_base.cc:1034] running on GCE node
W20260925 15:26:02.661307 12250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:02.661926 12244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:02.662285 12244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:02.663450 12244 hybrid_clock.cc:648] HybridClock initialized: now 1790349962663436 us; error 28 us; skew 500 ppm
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:02.666178 12244 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:26:02.666618 12244 webserver.cc:460] Webserver started at http://127.1.163.67:42795/ using document root <none> and password file <none>
I20260925 15:26:02.666875 12244 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:02.666922 12244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:02.667060 12244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:26:02.668195 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "6907114b3b4e4793ac05b36cc8307348"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "80bd7a2455bb1a60a96d20a72352ecb0"
server_key_iv: "9d64d6fba0273cfc6f3b40babcc74265"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.668658 12244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "6907114b3b4e4793ac05b36cc8307348"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "80bd7a2455bb1a60a96d20a72352ecb0"
server_key_iv: "9d64d6fba0273cfc6f3b40babcc74265"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.670320 12244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:26:02.671319 12259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.671536 12244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:26:02.671636 12244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "6907114b3b4e4793ac05b36cc8307348"
format_stamp: "Formatted at 2026-09-25 15:26:02 on dist-test-slave-hvv5"
server_key: "80bd7a2455bb1a60a96d20a72352ecb0"
server_key_iv: "9d64d6fba0273cfc6f3b40babcc74265"
server_key_version: "encryptionkey@0"
I20260925 15:26:02.671728 12244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:02.679126 12244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:02.681559 12244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:02.681730 12244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:02.681965 12244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:02.682279 12244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:02.682313 12244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.682372 12244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:02.682399 12244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:02.688498 12244 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:43321
I20260925 15:26:02.688583 12372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:43321 every 8 connection(s)
I20260925 15:26:02.688954 12244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:26:02.693408 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 12244
I20260925 15:26:02.693507 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:26:02.693768 1677 external_mini_cluster.cc:1428] Setting key aa97500e7f91304a83470a8d0978c69a
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[11869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:02.699488 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.689659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:35185 (local address 127.1.163.126:39837)
0925 15:26:02.689741 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.689744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.689992 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:26:02.690919 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.690920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.690963 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.691034 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:02.691654 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.692144 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.693237 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.693562 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.693645 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.696168 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:02.696170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:02.696170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:02.696177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:02.697966 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.698413 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.698414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.698415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.698468 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.698838 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.698838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.698838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.698966 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:02.699021 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.699338 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.699419 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:26:02.699661 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.689648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:39837 (local address 127.1.163.67:35185)
0925 15:26:02.689843 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.689880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.690793 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.691102 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.691106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.691377 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:02.691575 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.691580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.692228 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.692228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.693111 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.693114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.693650 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.693651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.693763 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.694497 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:02.694507 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:02.696084 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:02.698066 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.698073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.698075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.698341 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.698586 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:02.698587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:02.698588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:02.698764 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:02.699029 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:02.699034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:02.699164 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.699277 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.699422 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260925 15:26:02.700238 12373 heartbeater.cc:347] Connected to a master server at 127.1.163.126:39837
I20260925 15:26:02.700361 12373 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:02.700613 12373 heartbeater.cc:511] Master 127.1.163.126:39837 requested a full tablet report, sending...
I20260925 15:26:02.701076 11914 ts_manager.cc:194] Registered new tserver with Master: 6907114b3b4e4793ac05b36cc8307348 (127.1.163.67:43321)
I20260925 15:26:02.701447 11914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:35185
I20260925 15:26:02.705760 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:26:02.713959 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.707183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.1.163.126:39837)
0925 15:26:02.707282 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.707285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.707382 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:26:02.707468 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.707469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.707491 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.707576 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:02.708233 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.708827 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.709586 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.709783 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.709869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.711192 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:02.711194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:02.711194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:02.711202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:02.712706 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.713073 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.713075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.713075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.713135 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:02.713350 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:02.713351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:02.713351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:02.713511 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:02.713614 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.713769 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.713855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:26:02.715788 11914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:26:02.717648 11914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:26:02.724452 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.1.163.66:39231)
0925 15:26:02.720708 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.720711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.720721 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:26:02.720766 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.720773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.720786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.721162 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:02.721231 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.722347 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.723523 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724056 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724150 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724213 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.724322 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.724361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260925 15:26:02.724450 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39231 (local address 127.0.0.1:54024)
0925 15:26:02.720647 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.720657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.720698 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.720977 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.720981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.720983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:02.721072 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.721077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.722477 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.722478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.723434 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.723437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724148 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.724228 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724242 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.724309 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.724364 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":52,"threads_started":1}
I20260925 15:26:02.724769 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45801 (local address 127.0.0.1:35060)
0925 15:26:02.720761 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.720770 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.720819 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.721061 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.721061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.721063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:02.721109 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.721110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.722439 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.722440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.723526 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.723528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724485 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.724576 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724581 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.724638 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.724687 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":99,"threads_started":1}
I20260925 15:26:02.724884 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43321 (local address 127.0.0.1:46194)
0925 15:26:02.720495 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.720568 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.720641 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.721049 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.721050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.721052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:02.721113 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.721115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.722015 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.722016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.723246 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.723249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724283 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.724353 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724358 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.724407 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.724451 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260925 15:26:02.725072 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.1.163.65:45801)
0925 15:26:02.720678 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.720683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.720838 (+ 155us) server_negotiation.cc:407] Connection header received
0925 15:26:02.720888 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.720891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.720907 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.720980 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:02.721187 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.722310 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.723594 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724317 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724450 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.724917 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.724974 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":82,"threads_started":1}
I20260925 15:26:02.725298 12173 tablet_service.cc:1467] Processing CreateTablet for tablet fc994a41ae444d71b341a56ead86db94 (DEFAULT_TABLE table=test-table [id=c1a56ce87e5b4c2fb948f563cd969c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:02.725472 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.720483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.1.163.67:43321)
0925 15:26:02.720821 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.720826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.720835 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:26:02.720875 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.720877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.720893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.720973 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:02.721193 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.721908 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.723336 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.724101 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.724174 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.724244 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.724500 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.725390 (+ 890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":108,"threads_started":1}
I20260925 15:26:02.725695 12173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc994a41ae444d71b341a56ead86db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:02.726102 12307 tablet_service.cc:1467] Processing CreateTablet for tablet fc994a41ae444d71b341a56ead86db94 (DEFAULT_TABLE table=test-table [id=c1a56ce87e5b4c2fb948f563cd969c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:02.726408 12307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc994a41ae444d71b341a56ead86db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:02.728982 12038 tablet_service.cc:1467] Processing CreateTablet for tablet fc994a41ae444d71b341a56ead86db94 (DEFAULT_TABLE table=test-table [id=c1a56ce87e5b4c2fb948f563cd969c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:02.729336 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc994a41ae444d71b341a56ead86db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:02.729329 12390 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Bootstrap starting.
I20260925 15:26:02.729614 12391 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Bootstrap starting.
I20260925 15:26:02.730136 12390 tablet_bootstrap.cc:654] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:02.730163 12391 tablet_bootstrap.cc:654] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:02.730654 12391 log.cc:824] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:02.730716 12390 log.cc:824] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:02.731523 12391 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: No bootstrap required, opened a new log
I20260925 15:26:02.731600 12391 ts_tablet_manager.cc:1397] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:26:02.733016 12390 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: No bootstrap required, opened a new log
I20260925 15:26:02.733173 12394 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: Bootstrap starting.
I20260925 15:26:02.733233 12390 ts_tablet_manager.cc:1397] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:26:02.733629 12391 raft_consensus.cc:348] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.733800 12391 raft_consensus.cc:374] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:02.733842 12391 raft_consensus.cc:729] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6907114b3b4e4793ac05b36cc8307348, State: Initialized, Role: FOLLOWER
I20260925 15:26:02.733842 12394 tablet_bootstrap.cc:654] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:02.733956 12391 consensus_queue.cc:260] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.734273 12391 ts_tablet_manager.cc:1428] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:26:02.734305 12373 heartbeater.cc:503] Master 127.1.163.126:39837 was elected leader, sending a full tablet report...
I20260925 15:26:02.734390 12394 log.cc:824] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:02.735266 12394 tablet_bootstrap.cc:492] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: No bootstrap required, opened a new log
I20260925 15:26:02.735340 12394 ts_tablet_manager.cc:1397] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:26:02.735359 12390 raft_consensus.cc:348] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.735527 12390 raft_consensus.cc:374] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:02.735565 12390 raft_consensus.cc:729] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a2249cb1934ce2813f94efc2dd0e5c, State: Initialized, Role: FOLLOWER
I20260925 15:26:02.735684 12390 consensus_queue.cc:260] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.735965 12390 ts_tablet_manager.cc:1428] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:26:02.735987 12239 heartbeater.cc:503] Master 127.1.163.126:39837 was elected leader, sending a full tablet report...
I20260925 15:26:02.737354 12394 raft_consensus.cc:348] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.737494 12394 raft_consensus.cc:374] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:02.737516 12394 raft_consensus.cc:729] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7672261426c480bbc0754ed6dc8692b, State: Initialized, Role: FOLLOWER
I20260925 15:26:02.737624 12394 consensus_queue.cc:260] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.737926 12394 ts_tablet_manager.cc:1428] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:26:02.737959 12105 heartbeater.cc:503] Master 127.1.163.126:39837 was elected leader, sending a full tablet report...
W20260925 15:26:02.788187 12240 tablet.cc:2369] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:26:02.796204 12395 raft_consensus.cc:482] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:26:02.796337 12395 raft_consensus.cc:504] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.796742 12395 leader_election.cc:288] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7672261426c480bbc0754ed6dc8692b (127.1.163.65:45801), 38a2249cb1934ce2813f94efc2dd0e5c (127.1.163.66:39231)
I20260925 15:26:02.799780 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.797184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:37001 (local address 127.1.163.66:39231)
0925 15:26:02.797225 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.797227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.797236 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:26:02.797265 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.797266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.797276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.797331 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:02.797491 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.797951 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799031 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.799416 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.799515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.799665 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.799704 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:26:02.799798 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.797014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40913 (local address 127.1.163.65:45801)
0925 15:26:02.797061 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.797064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.797075 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:26:02.797157 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.797158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.797168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.797212 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:02.797390 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.797962 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799023 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.799428 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799520 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.799529 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.799683 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.799730 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:26:02.799798 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.796909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45801 (local address 127.1.163.67:40913)
0925 15:26:02.797017 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.797028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.797068 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.797260 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.797260 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.797262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:02.797327 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.797329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.798055 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.798056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.798921 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.798923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.799517 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.799597 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.799600 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.799664 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.799711 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":69}
I20260925 15:26:02.799808 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.796929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39231 (local address 127.1.163.67:37001)
0925 15:26:02.797141 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.797152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:02.797213 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:02.797334 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:02.797335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:02.797336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:02.797413 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.797414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.798043 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.798044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.798921 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:02.798924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.799504 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.799505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:02.799578 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.799582 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:26:02.799659 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:26:02.799703 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":52,"thread_start_us":120,"threads_started":1}
I20260925 15:26:02.800247 12059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc994a41ae444d71b341a56ead86db94" candidate_uuid: "6907114b3b4e4793ac05b36cc8307348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7672261426c480bbc0754ed6dc8692b" is_pre_election: true
I20260925 15:26:02.800395 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc994a41ae444d71b341a56ead86db94" candidate_uuid: "6907114b3b4e4793ac05b36cc8307348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" is_pre_election: true
I20260925 15:26:02.800447 12059 raft_consensus.cc:2454] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6907114b3b4e4793ac05b36cc8307348 in term 0.
I20260925 15:26:02.800527 12193 raft_consensus.cc:2454] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6907114b3b4e4793ac05b36cc8307348 in term 0.
I20260925 15:26:02.800720 12262 leader_election.cc:302] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6907114b3b4e4793ac05b36cc8307348, d7672261426c480bbc0754ed6dc8692b; no voters:
I20260925 15:26:02.800907 12395 raft_consensus.cc:2781] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:26:02.800967 12395 raft_consensus.cc:482] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:26:02.800995 12395 raft_consensus.cc:3037] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:02.801736 12395 raft_consensus.cc:504] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.801892 12395 leader_election.cc:288] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [CANDIDATE]: Term 1 election: Requested vote from peers d7672261426c480bbc0754ed6dc8692b (127.1.163.65:45801), 38a2249cb1934ce2813f94efc2dd0e5c (127.1.163.66:39231)
I20260925 15:26:02.802026 12059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc994a41ae444d71b341a56ead86db94" candidate_uuid: "6907114b3b4e4793ac05b36cc8307348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7672261426c480bbc0754ed6dc8692b"
I20260925 15:26:02.802115 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc994a41ae444d71b341a56ead86db94" candidate_uuid: "6907114b3b4e4793ac05b36cc8307348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a2249cb1934ce2813f94efc2dd0e5c"
I20260925 15:26:02.802111 12059 raft_consensus.cc:3037] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:02.802186 12193 raft_consensus.cc:3037] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:02.803032 12059 raft_consensus.cc:2454] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6907114b3b4e4793ac05b36cc8307348 in term 1.
I20260925 15:26:02.803033 12193 raft_consensus.cc:2454] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6907114b3b4e4793ac05b36cc8307348 in term 1.
I20260925 15:26:02.803246 12262 leader_election.cc:302] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6907114b3b4e4793ac05b36cc8307348, d7672261426c480bbc0754ed6dc8692b; no voters:
I20260925 15:26:02.803349 12395 raft_consensus.cc:2781] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:26:02.803503 12395 raft_consensus.cc:686] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 1 LEADER]: Becoming Leader. State: Replica: 6907114b3b4e4793ac05b36cc8307348, State: Running, Role: LEADER
I20260925 15:26:02.803609 12395 consensus_queue.cc:237] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } }
I20260925 15:26:02.804400 11914 catalog_manager.cc:5495] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6907114b3b4e4793ac05b36cc8307348 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "6907114b3b4e4793ac05b36cc8307348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6907114b3b4e4793ac05b36cc8307348" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 } health_report { overall_health: UNKNOWN } } }
I20260925 15:26:02.816987 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.814755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.1.163.67:43321)
0925 15:26:02.814799 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:02.814801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:02.814809 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:26:02.814889 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:02.814890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:02.814907 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:02.814959 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:26:02.815082 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.815542 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.816254 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:02.816440 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:02.816494 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:02.816644 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:26:02.816741 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:26:02.816798 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:02.816833 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 15:26:02.816852 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:26:02.818841 12193 raft_consensus.cc:1261] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 1 FOLLOWER]: Refusing update from remote peer 6907114b3b4e4793ac05b36cc8307348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:26:02.818841 12059 raft_consensus.cc:1261] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 1 FOLLOWER]: Refusing update from remote peer 6907114b3b4e4793ac05b36cc8307348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:26:02.819237 12395 consensus_queue.cc:1035] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a2249cb1934ce2813f94efc2dd0e5c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:26:02.819391 12400 consensus_queue.cc:1035] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7672261426c480bbc0754ed6dc8692b" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:26:02.822108 12404 mvcc.cc:205] Tried to move back new op lower bound from 7333273447703183360 to 7333273447644266496. Current Snapshot: MvccSnapshot[applied={T|T < 7333273447703183360}]
I20260925 15:26:02.823994 11914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:26:02.824115 11914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:26:02.825110 11914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302640e22a0f4c1980e6281a9809f7fd: Sending DeleteTablet for 3 replicas of tablet fc994a41ae444d71b341a56ead86db94
I20260925 15:26:02.825517 12307 tablet_service.cc:1514] Processing DeleteTablet for tablet fc994a41ae444d71b341a56ead86db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:02 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:46194
I20260925 15:26:02.825538 12173 tablet_service.cc:1514] Processing DeleteTablet for tablet fc994a41ae444d71b341a56ead86db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:02 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:54024
I20260925 15:26:02.825541 12038 tablet_service.cc:1514] Processing DeleteTablet for tablet fc994a41ae444d71b341a56ead86db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:02 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:35060
I20260925 15:26:02.825762 12414 tablet_replica.cc:330] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c: stopping tablet replica
I20260925 15:26:02.825871 12414 raft_consensus.cc:2229] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:26:02.825903 12415 tablet_replica.cc:330] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b: stopping tablet replica
I20260925 15:26:02.825997 12415 raft_consensus.cc:2229] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:26:02.826092 12415 raft_consensus.cc:2258] T fc994a41ae444d71b341a56ead86db94 P d7672261426c480bbc0754ed6dc8692b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:26:02.826189 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11976
I20260925 15:26:02.826261 12413 tablet_replica.cc:330] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: stopping tablet replica
I20260925 15:26:02.827191 12413 raft_consensus.cc:2229] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:26:02.827306 12413 raft_consensus.cc:2258] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:26:02.827812 12413 ts_tablet_manager.cc:1905] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:26:02.830001 12413 ts_tablet_manager.cc:1918] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:26:02.830088 12413 log.cc:1196] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/fc994a41ae444d71b341a56ead86db94
I20260925 15:26:02.830400 12413 ts_tablet_manager.cc:1939] T fc994a41ae444d71b341a56ead86db94 P 6907114b3b4e4793ac05b36cc8307348: Deleting consensus metadata
I20260925 15:26:02.830951 11901 catalog_manager.cc:4841] TS 6907114b3b4e4793ac05b36cc8307348 (127.1.163.67:43321): tablet fc994a41ae444d71b341a56ead86db94 (table test-table [id=c1a56ce87e5b4c2fb948f563cd969c58]) successfully deleted
W20260925 15:26:02.833154 11900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:45801 (error 108)
I20260925 15:26:02.825975 12414 raft_consensus.cc:2258] T fc994a41ae444d71b341a56ead86db94 P 38a2249cb1934ce2813f94efc2dd0e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:26:02.833483 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 12110
I20260925 15:26:02.834101 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.833589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45801 (local address 127.0.0.1:35074)
0925 15:26:02.834024 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.834066 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:45801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410}
W20260925 15:26:02.834944 11900 catalog_manager.cc:4584] TS d7672261426c480bbc0754ed6dc8692b (127.1.163.65:45801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc994a41ae444d71b341a56ead86db94: Network error: Client connection negotiation failed: client connection to 127.1.163.65:45801: connect: Connection refused (error 111)
I20260925 15:26:02.839246 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 12244
I20260925 15:26:02.839401 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:02.839261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39231 (local address 127.0.0.1:54036)
0925 15:26:02.839324 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:02.839354 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.66:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:26:02.844192 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 11882
2026-09-25T15:26:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1988 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[12419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[12419](info): set up 2 sockets
Sep 25 15:26:02 dist-test-slave-hvv5 krb5kdc[12419](info): commencing operation
krb5kdc: starting...
W20260925 15:26:03.893594 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:26:03 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:26:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangeradmin_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangeradmin_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangerlookup_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangerlookup_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
I20260925 15:26:04.071803 1677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk86Bl6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/postgres -l logfile start
2026-09-25 15:26:06.976 UTC [12449] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:26:06.976 UTC [12449] LOG: listening on IPv4 address "127.1.163.84", port 59443
2026-09-25 15:26:06.981 UTC [12449] LOG: listening on Unix socket "/tmp/.s.PGSQL.59443"
2026-09-25 15:26:07.018 UTC [12452] LOG: database system was shut down at 2026-09-25 15:26:05 UTC
2026-09-25 15:26:07.022 UTC [12449] LOG: database system is ready to accept connections
I20260925 15:26:07.961568 1677 mini_postgres.cc:96] Postgres bound to 59443
2026-09-25 15:26:07.968 UTC [12460] FATAL: database "slave" does not exist
127.1.163.84:59443 - accepting connections
I20260925 15:26:07.968564 1677 mini_ranger.cc:161] Starting Ranger...
I20260925 15:26:07.983498 1677 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:26:08.583283 1677 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:26:08.583382 1677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin
2026-09-25 15:26:08,892 [I] DB FLAVOR :POSTGRES
2026-09-25 15:26:08,893 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:26:08,893 [I] Checking connection..
2026-09-25 15:26:08,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:26:09,179 [I] Checking connection passed.
2026-09-25 15:26:09,179 [I] --------- Verifying version history table ---------
2026-09-25 15:26:09,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:09,435 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:26:09,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:09,681 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:26:09,681 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:26:09,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:26:09.950 UTC [12524] WARNING: there is no transaction in progress
2026-09-25 15:26:09,957 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:26:09,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:10,203 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:26:10,204 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:26:10,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:26:10,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:26:10,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvv5.c.gcp-upstream.internal','N') ;"
2026-09-25 15:26:10,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:26:11,184 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:26:11,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:26:11,438 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:26:11,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:26:11,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:26:11,951 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:26:11,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:26:13.054 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.074 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.093 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.111 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.129 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.149 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.214 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.222 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.240 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.253 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.265 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.282 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.294 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.302 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.312 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:13.320 UTC [12632] WARNING: there is no transaction in progress
2026-09-25 15:26:14,081 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:26:14,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:26:14,334 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:26:14,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:26:14,605 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:26:14,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:26:14,852 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:26:14,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:26:15,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:26:15,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvv5.c.gcp-upstream.internal';"
2026-09-25 15:26:15,594 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:26:15,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:26:15,835 [I] DB_PATCHES have already been applied
I20260925 15:26:15.841277 1677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk86Bl6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:26:15.841349 1677 mini_ranger.cc:193] Using host: 127.1.163.84
I20260925 15:26:15.844527 1677 mini_ranger.cc:227] Ranger admin URL: http://127.1.163.84:33165
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:26:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:26:16.994246 1677 mini_ranger.cc:160] Time spent starting Ranger: real 9.026s user 0.005s sys 0.004s
Sep 25, 2026 3:26:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:26:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:26:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:26:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:26:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:26:38.599113 1677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangerkms_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/rangerkms_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.163.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.163.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:26:38 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349998, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:26:38.696878 1677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:26:38.709832 1677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:26:39.925815 1677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:26:39.925911 1677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms
I20260925 15:26:39.925935 1677 mini_ranger_kms.cc:227] Using postgres at 127.1.163.84:59443
2026-09-25 15:26:40,161 [I] DB FLAVOR :POSTGRES
2026-09-25 15:26:40,161 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:26:40,161 [I] Checking connection
2026-09-25 15:26:40,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:26:40,418 [I] connection success
2026-09-25 15:26:40,418 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:26:40,419 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:26:40,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:26:40,667 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:26:40,667 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:26:40,668 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:26:40,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:59443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:26:40,989 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:26:41.533546 1677 mini_ranger_kms.cc:310] Created kms service
I20260925 15:26:41.704931 1677 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:26:41.756932 1677 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:26:41 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349998, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:26:42.201634 1677 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:26:42.201758 1677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:26:42.201781 1677 mini_ranger_kms.cc:240] Using host: 127.1.163.84
I20260925 15:26:42.205515 1677 mini_ranger_kms.cc:276] Ranger KMS PID: 12812
I20260925 15:26:42.205650 1677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.163.84:50997
Sep 25, 2026 3:26:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:26:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:26:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:26:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:26:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:26:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:26:44.524708 1677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.828s user 0.009s sys 0.204s
I20260925 15:26:44.524811 1677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:26:44.524883 1677 mini_ranger_kms.cc:401] 127.1.163.84:50997/kms/v1/keys
Sep 25 15:26:44 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790350004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:26:45 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:26:45 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:26:48 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:26:48.062402 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41461
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:41461
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:50997/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:48.148824 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:48.149063 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:48.149087 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:48.150331 12863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:26:48.150382 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:48.150398 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:48.150409 12863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:26:48.150422 12863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:26:48.152294 12863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.163.84:50997/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:41461
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:48.152642 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:48.152935 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:48.158921 12871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:48.158931 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:48.158968 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:48.160069 12863 server_base.cc:1034] running on GCE node
I20260925 15:26:48.160390 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:48.160789 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:48.161969 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790350008161949 us; error 40 us; skew 500 ppm
Sep 25 15:26:48 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350008, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:48.170519 12863 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:26:48.171082 12863 webserver.cc:460] Webserver started at http://127.1.163.126:35365/ using document root <none> and password file <none>
I20260925 15:26:48.171355 12863 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:48.171418 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:48.171574 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:26:48 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350008, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:26:48.230126 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "ea623bc943b54e95bafe53601ffaa38b"
format_stamp: "Formatted at 2026-09-25 15:26:48 on dist-test-slave-hvv5"
server_key: "189a3054a0d68bf31348d409a762e598"
server_key_iv: "ecc4151f499f87c47e730f82c20086be"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:48.230604 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "ea623bc943b54e95bafe53601ffaa38b"
format_stamp: "Formatted at 2026-09-25 15:26:48 on dist-test-slave-hvv5"
server_key: "189a3054a0d68bf31348d409a762e598"
server_key_iv: "ecc4151f499f87c47e730f82c20086be"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:48.232519 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:26:48.245020 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:48.245210 12863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:26:48.245321 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "ea623bc943b54e95bafe53601ffaa38b"
format_stamp: "Formatted at 2026-09-25 15:26:48 on dist-test-slave-hvv5"
server_key: "189a3054a0d68bf31348d409a762e598"
server_key_iv: "ecc4151f499f87c47e730f82c20086be"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:48.245438 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:48.254822 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:48.257079 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:48.257248 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:48.261787 12863 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:41461
I20260925 15:26:48.261806 12930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:41461 every 8 connection(s)
I20260925 15:26:48.262259 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:26:48.263042 12931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:48.265589 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: Bootstrap starting.
I20260925 15:26:48.266224 12931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:48.266554 12931 log.cc:824] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:48.267359 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: No bootstrap required, opened a new log
I20260925 15:26:48.269202 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 12863
I20260925 15:26:48.269304 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:26:48.269497 1677 external_mini_cluster.cc:1428] Setting key 32b01a7e8afca1d93962fe238d48cfb2
I20260925 15:26:48.270001 12931 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } }
I20260925 15:26:48.270308 12931 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:48.270385 12931 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea623bc943b54e95bafe53601ffaa38b, State: Initialized, Role: FOLLOWER
I20260925 15:26:48.270543 12931 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } }
I20260925 15:26:48.270720 12931 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:26:48.270776 12931 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:26:48.270848 12931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:48.271646 12931 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } }
I20260925 15:26:48.271775 12931 leader_election.cc:302] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea623bc943b54e95bafe53601ffaa38b; no voters:
I20260925 15:26:48.271984 12931 leader_election.cc:288] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:26:48.272011 12936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:26:48.272228 12936 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [term 1 LEADER]: Becoming Leader. State: Replica: ea623bc943b54e95bafe53601ffaa38b, State: Running, Role: LEADER
I20260925 15:26:48.272364 12931 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:26:48.272354 12936 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } }
I20260925 15:26:48.274322 12931 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:26:48.274293 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea623bc943b54e95bafe53601ffaa38b. Latest consensus state: current_term: 1 leader_uuid: "ea623bc943b54e95bafe53601ffaa38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } } }
I20260925 15:26:48.274519 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [sys.catalog]: This master's current role is: LEADER
I20260925 15:26:48.274744 12936 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea623bc943b54e95bafe53601ffaa38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea623bc943b54e95bafe53601ffaa38b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41461 } } }
I20260925 15:26:48.274863 12936 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b [sys.catalog]: This master's current role is: LEADER
Sep 25 15:26:48 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:48.277822 12951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:26:48.278909 12951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:26:48.281157 12951 catalog_manager.cc:1347] Generated new cluster ID: 5f5c29b8335d448098d6ff30c55f797d
I20260925 15:26:48.281209 12951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:26:48.282924 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:48.270053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.1.163.126:41461)
0925 15:26:48.270254 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:48.270263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:48.270295 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:26:48.270424 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:48.270432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:48.270489 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:48.270641 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:48.271440 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:48.272815 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:48.273973 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:48.274621 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:48.274734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:48.277865 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:48.277908 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:48.277908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:48.277957 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:48.280468 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:48.280940 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:48.280943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:48.280944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:48.281042 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:48.281231 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:48.281232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:48.281232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:48.281609 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:48.281747 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:48.282480 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 15:26:48.282636 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260925 15:26:48.296836 12951 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:26:48.297896 12951 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:26:48.304576 12951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: Generated new TSK 0
I20260925 15:26:48.304894 12951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:26:49.053357 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:50997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:49.167244 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:49.167495 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:49.167541 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:49.169000 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:49.169064 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:49.169118 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:26:49.171231 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.163.84:50997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:49.171499 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:49.171823 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:49.179807 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:49.179865 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:49.179913 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:49.180438 12976 server_base.cc:1034] running on GCE node
I20260925 15:26:49.180677 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:49.180968 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:49.182163 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790350009182147 us; error 27 us; skew 500 ppm
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:49.185815 12976 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:26:49.186319 12976 webserver.cc:460] Webserver started at http://127.1.163.65:42033/ using document root <none> and password file <none>
I20260925 15:26:49.186558 12976 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:49.186630 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:49.186791 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:26:49.201439 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "ab22f721d60c41b69d291d5683812803"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "251bd7da1fe73ab105b1272825ed7c10"
server_key_iv: "6dcf2a8781a0fed9d269e7695f4d6fb9"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.201983 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "ab22f721d60c41b69d291d5683812803"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "251bd7da1fe73ab105b1272825ed7c10"
server_key_iv: "6dcf2a8781a0fed9d269e7695f4d6fb9"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.204056 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:26:49.220311 12993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.220510 12976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:26:49.220635 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "ab22f721d60c41b69d291d5683812803"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "251bd7da1fe73ab105b1272825ed7c10"
server_key_iv: "6dcf2a8781a0fed9d269e7695f4d6fb9"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.220755 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:49.233120 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:49.235685 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:49.235875 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:49.236084 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:49.236449 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:49.236509 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.236574 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:49.236593 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.242816 12976 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:41077
I20260925 15:26:49.243251 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:26:49.246493 13106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:41077 every 8 connection(s)
I20260925 15:26:49.251490 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 12976
I20260925 15:26:49.251600 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:26:49.251856 1677 external_mini_cluster.cc:1428] Setting key 0f31fdf035cd109b2f9b0d020fc7563a
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:49.262377 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.247292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:44165 (local address 127.1.163.126:41461)
0925 15:26:49.247461 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.247463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.248668 (+ 1205us) server_negotiation.cc:407] Connection header received
0925 15:26:49.249387 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.249388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.249403 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.249450 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:49.250368 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.251244 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.255318 (+ 4074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.255629 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.255675 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.258516 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:49.258518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:49.258518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:49.258528 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:49.260497 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.261094 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.261096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.261097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.261165 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.261489 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.261491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.261491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.261645 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:49.261708 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.262133 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.262236 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260925 15:26:49.263231 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.247252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41461 (local address 127.1.163.65:44165)
0925 15:26:49.248488 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.248568 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.249255 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.249684 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.249688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.249960 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:49.250248 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.250254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.252139 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.252140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.255172 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.255178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.255797 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.255798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.255961 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.256688 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:49.256701 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:49.258371 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:49.260637 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.260645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.260648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.260993 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.261243 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.261243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.261244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.261392 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.261748 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:49.261755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:49.261955 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.262818 (+ 863us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.262999 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":94,"threads_started":1}
I20260925 15:26:49.265676 13107 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41461
I20260925 15:26:49.265805 13107 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:49.266033 13107 heartbeater.cc:511] Master 127.1.163.126:41461 requested a full tablet report, sending...
I20260925 15:26:49.266594 12895 ts_manager.cc:194] Registered new tserver with Master: ab22f721d60c41b69d291d5683812803 (127.1.163.65:41077)
I20260925 15:26:49.267612 12895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:44165
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:26:49.300829 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:50997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:49.419988 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:49.420228 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:49.420269 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:49.421545 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:49.421617 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:49.421664 13113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:26:49.423434 13113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.163.84:50997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:49.423725 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:49.424003 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:49.430619 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:49.430974 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:49.432992 13113 server_base.cc:1034] running on GCE node
W20260925 15:26:49.433197 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:49.433422 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:49.433709 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:49.434882 13113 hybrid_clock.cc:648] HybridClock initialized: now 1790350009434860 us; error 35 us; skew 500 ppm
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:49.437645 13113 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:26:49.438198 13113 webserver.cc:460] Webserver started at http://127.1.163.66:40285/ using document root <none> and password file <none>
I20260925 15:26:49.438489 13113 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:49.438562 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:49.438721 13113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:49.472921 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "f05c3daf30a7f36f62955e7cf607f3b3"
server_key_iv: "d6112cd39e81ea0e6c8cf85ceab9f1cf"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.473549 13113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "f05c3daf30a7f36f62955e7cf607f3b3"
server_key_iv: "d6112cd39e81ea0e6c8cf85ceab9f1cf"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.482437 13113 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260925 15:26:49.501616 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.501936 13113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:26:49.502034 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "f05c3daf30a7f36f62955e7cf607f3b3"
server_key_iv: "d6112cd39e81ea0e6c8cf85ceab9f1cf"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.502127 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:49.513827 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:49.516330 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:49.516474 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:49.516754 13113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:49.517112 13113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:49.517159 13113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.517194 13113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:49.517212 13113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.524592 13113 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:40009
I20260925 15:26:49.525050 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:26:49.526049 13244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:40009 every 8 connection(s)
I20260925 15:26:49.528478 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 13113
I20260925 15:26:49.528586 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:26:49.528774 1677 external_mini_cluster.cc:1428] Setting key da7617851a8dd94548bf7456dc2dd999
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:49.540515 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.526961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:48407 (local address 127.1.163.126:41461)
0925 15:26:49.527010 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.527012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.527348 (+ 336us) server_negotiation.cc:407] Connection header received
0925 15:26:49.528053 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.528055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.528072 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.528103 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:49.529810 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.530611 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.533846 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.534181 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.534226 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.537015 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:49.537017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:49.537018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:49.537026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:49.538794 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.539332 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.539334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.539334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.539395 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.539703 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.539704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.539704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.539853 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:49.539906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.540315 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.540415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:26:49.541365 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.526677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41461 (local address 127.1.163.66:48407)
0925 15:26:49.527208 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.527251 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.527896 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.529059 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.529066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.529351 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:49.529676 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.529685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.532381 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.532382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.533730 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.533733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.534337 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.534338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.534501 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.535147 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:49.535158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:49.536893 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:49.538904 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.538913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.538915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.539223 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.539463 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.539463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.539463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.539624 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.539940 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:49.539947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:49.540127 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.540921 (+ 794us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.541104 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":88,"threads_started":1}
I20260925 15:26:49.542066 13245 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41461
I20260925 15:26:49.542163 13245 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:49.542369 13245 heartbeater.cc:511] Master 127.1.163.126:41461 requested a full tablet report, sending...
I20260925 15:26:49.542739 12895 ts_manager.cc:194] Registered new tserver with Master: 5036d6ac7ddb4c1cad246a9e9a96a2d1 (127.1.163.66:40009)
I20260925 15:26:49.543228 12895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:48407
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:26:49.577255 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:50997/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:26:49.702644 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:26:49.703059 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:26:49.703140 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:26:49.705461 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:26:49.705600 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:26:49.705681 13250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:26:49.707782 13250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.163.84:50997/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:26:49.708140 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:26:49.708439 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:26:49.715596 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:49.716101 13250 server_base.cc:1034] running on GCE node
W20260925 15:26:49.716246 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:26:49.720774 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:26:49.724964 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:26:49.725399 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:26:49.726632 13250 hybrid_clock.cc:648] HybridClock initialized: now 1790350009726559 us; error 103 us; skew 500 ppm
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:26:49.729645 13250 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:26:49.730192 13250 webserver.cc:460] Webserver started at http://127.1.163.67:46139/ using document root <none> and password file <none>
I20260925 15:26:49.730496 13250 fs_manager.cc:359] Metadata directory not provided
I20260925 15:26:49.730577 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:26:49.730748 13250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:26:49.745095 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "29808b0f05014c96b656191844acda57"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "bf0c485c96b8eb7c7613e1b865c52516"
server_key_iv: "a4239dd3a532807c0236256cf7273fb8"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.745651 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "29808b0f05014c96b656191844acda57"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "bf0c485c96b8eb7c7613e1b865c52516"
server_key_iv: "a4239dd3a532807c0236256cf7273fb8"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.747308 13250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:26:49.779522 13265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.781961 13250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:26:49.782111 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "29808b0f05014c96b656191844acda57"
format_stamp: "Formatted at 2026-09-25 15:26:49 on dist-test-slave-hvv5"
server_key: "bf0c485c96b8eb7c7613e1b865c52516"
server_key_iv: "a4239dd3a532807c0236256cf7273fb8"
server_key_version: "kuduclusterkey@0"
I20260925 15:26:49.782234 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:26:49.809170 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:26:49.811715 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:26:49.811877 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:26:49.812151 13250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:26:49.812609 13250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:26:49.812664 13250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.812700 13250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:26:49.812724 13250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:26:49.819599 13250 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:35217
I20260925 15:26:49.820066 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:26:49.822803 13378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:35217 every 8 connection(s)
I20260925 15:26:49.827749 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 13250
I20260925 15:26:49.827852 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:26:49.828063 1677 external_mini_cluster.cc:1428] Setting key 95266276bc92c1565c39cb924fef0f3c
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:26:49.837212 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.823767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:45911 (local address 127.1.163.126:41461)
0925 15:26:49.823819 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.823822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.824199 (+ 377us) server_negotiation.cc:407] Connection header received
0925 15:26:49.825021 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.825024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.825038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.825195 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:49.826261 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.827104 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.830292 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.830619 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.830665 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.833588 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:49.833590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:49.833590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:49.833600 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:49.835419 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.835992 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.835994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.835995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.836062 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.836397 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.836398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.836398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.836570 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:49.836629 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.837013 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.837108 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:26:49.838071 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.823444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41461 (local address 127.1.163.67:45911)
0925 15:26:49.824055 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.824105 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.824882 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.825556 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.825561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.825829 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:26:49.826127 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.826135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.827259 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.827260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.830177 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.830181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.830767 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.830768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.830927 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.831628 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:26:49.831639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:26:49.833451 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:26:49.835538 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.835547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.835550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.835888 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.836138 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:26:49.836139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:26:49.836139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:26:49.836298 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:26:49.836664 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:26:49.836670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:26:49.836847 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.837621 (+ 774us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.837820 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":102,"threads_started":1}
I20260925 15:26:49.838871 13379 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41461
I20260925 15:26:49.838972 13379 heartbeater.cc:464] Registering TS with master...
I20260925 15:26:49.839205 13379 heartbeater.cc:511] Master 127.1.163.126:41461 requested a full tablet report, sending...
I20260925 15:26:49.840263 12895 ts_manager.cc:194] Registered new tserver with Master: 29808b0f05014c96b656191844acda57 (127.1.163.67:35217)
I20260925 15:26:49.840864 12895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:45911
I20260925 15:26:49.850945 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:26:49.861502 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.852722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.1.163.126:41461)
0925 15:26:49.852772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.852774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.853151 (+ 377us) server_negotiation.cc:407] Connection header received
0925 15:26:49.853264 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.853265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.853278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.853309 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:26:49.854179 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.854980 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.856105 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.856401 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.856452 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.858294 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:26:49.858296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:26:49.858297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:26:49.858305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:26:49.860121 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.860600 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.860601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.860602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.860664 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:26:49.860878 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:26:49.860879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:26:49.860879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:26:49.861021 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:26:49.861098 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.861310 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.861407 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:26:49.863951 12895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:26:49.866549 12895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:26:49.877043 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:35217 (local address 127.0.0.1:58710)
0925 15:26:49.871581 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.871588 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.871623 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.871949 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.871954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.871957 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:49.872036 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.872038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.873820 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.873821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.875554 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.875558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.876705 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.876707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.876792 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.876808 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.876883 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.876943 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":51,"threads_started":1}
I20260925 15:26:49.877514 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.1.163.67:35217)
0925 15:26:49.871419 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.871424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.871669 (+ 245us) server_negotiation.cc:407] Connection header received
0925 15:26:49.871713 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.871715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.871771 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.871844 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:49.872125 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.873717 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.875681 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.876481 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.876578 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.876666 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.877384 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.877432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":82,"threads_started":1}
I20260925 15:26:49.878670 13313 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c76549f74192964d06e53d5b6693 (DEFAULT_TABLE table=test-table [id=48ee271e5a0d41f081aef75c47eda5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:49.879074 13313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c76549f74192964d06e53d5b6693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:49.888468 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:40009 (local address 127.0.0.1:32882)
0925 15:26:49.871467 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.871476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.871512 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.884867 (+ 13355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.884868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.884871 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:49.884921 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.884923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.886193 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.886194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.887287 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.887290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.888245 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.888246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.888309 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.888314 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.888343 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.888384 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":65,"threads_started":1}
I20260925 15:26:49.888937 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.1.163.66:40009)
0925 15:26:49.884654 (+ 14235us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.884659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.884673 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:26:49.884715 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.884718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.884734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.884989 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:49.885063 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.886090 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.887409 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.888145 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.888698 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.888800 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.888828 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.888866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14188,"thread_start_us":70,"threads_started":1}
I20260925 15:26:49.889232 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:41077 (local address 127.0.0.1:58002)
0925 15:26:49.871149 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.871195 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.871257 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.883922 (+ 12665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.883924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.883927 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:49.883989 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.883991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.885760 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.885761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.887086 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.887090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.888261 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.888262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.888975 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.888980 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.889109 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.889156 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":75,"threads_started":1}
I20260925 15:26:49.889729 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.870187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.1.163.65:41077)
0925 15:26:49.883716 (+ 13529us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.883722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.883736 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:26:49.883776 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.883778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.883800 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.884426 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:49.884516 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.885641 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.887197 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.888151 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.889509 (+ 1358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.889575 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.889603 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.889648 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13486,"thread_start_us":67,"threads_started":1}
I20260925 15:26:49.889717 13179 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c76549f74192964d06e53d5b6693 (DEFAULT_TABLE table=test-table [id=48ee271e5a0d41f081aef75c47eda5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:49.890132 13179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c76549f74192964d06e53d5b6693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:49.890496 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 7285c76549f74192964d06e53d5b6693 (DEFAULT_TABLE table=test-table [id=48ee271e5a0d41f081aef75c47eda5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:26:49.890789 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7285c76549f74192964d06e53d5b6693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:26:49.894757 13398 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Bootstrap starting.
I20260925 15:26:49.895951 13398 tablet_bootstrap.cc:654] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:49.896380 13398 log.cc:824] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:49.896832 13400 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Bootstrap starting.
I20260925 15:26:49.897625 13400 tablet_bootstrap.cc:654] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:49.898018 13400 log.cc:824] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:49.898976 13398 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: No bootstrap required, opened a new log
I20260925 15:26:49.899101 13398 ts_tablet_manager.cc:1397] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:26:49.901877 13398 raft_consensus.cc:348] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.902209 13398 raft_consensus.cc:374] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:49.902311 13398 raft_consensus.cc:729] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29808b0f05014c96b656191844acda57, State: Initialized, Role: FOLLOWER
I20260925 15:26:49.902545 13398 consensus_queue.cc:260] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.902956 13398 ts_tablet_manager.cc:1428] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:26:49.903131 13379 heartbeater.cc:503] Master 127.1.163.126:41461 was elected leader, sending a full tablet report...
I20260925 15:26:49.904946 13400 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: No bootstrap required, opened a new log
I20260925 15:26:49.905081 13400 ts_tablet_manager.cc:1397] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 15:26:49.906258 13399 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Bootstrap starting.
I20260925 15:26:49.907491 13399 tablet_bootstrap.cc:654] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:26:49.907838 13399 log.cc:824] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:26:49.908114 13400 raft_consensus.cc:348] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.908371 13400 raft_consensus.cc:374] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:49.908461 13400 raft_consensus.cc:729] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab22f721d60c41b69d291d5683812803, State: Initialized, Role: FOLLOWER
I20260925 15:26:49.908644 13400 consensus_queue.cc:260] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.908959 13400 ts_tablet_manager.cc:1428] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:26:49.908986 13107 heartbeater.cc:503] Master 127.1.163.126:41461 was elected leader, sending a full tablet report...
I20260925 15:26:49.913447 13399 tablet_bootstrap.cc:492] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: No bootstrap required, opened a new log
I20260925 15:26:49.913558 13399 ts_tablet_manager.cc:1397] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 15:26:49.915587 13399 raft_consensus.cc:348] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.915827 13399 raft_consensus.cc:374] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:26:49.915894 13399 raft_consensus.cc:729] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5036d6ac7ddb4c1cad246a9e9a96a2d1, State: Initialized, Role: FOLLOWER
I20260925 15:26:49.916062 13399 consensus_queue.cc:260] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.916402 13399 ts_tablet_manager.cc:1428] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:26:49.916450 13245 heartbeater.cc:503] Master 127.1.163.126:41461 was elected leader, sending a full tablet report...
Sep 25 15:26:49 dist-test-slave-hvv5 krb5kdc[12419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:26:49.989081 13403 raft_consensus.cc:482] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:26:49.989244 13403 raft_consensus.cc:504] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.989672 13403 leader_election.cc:288] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5036d6ac7ddb4c1cad246a9e9a96a2d1 (127.1.163.66:40009), ab22f721d60c41b69d291d5683812803 (127.1.163.65:41077)
I20260925 15:26:49.994266 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.989912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:40009 (local address 127.1.163.67:46423)
0925 15:26:49.990050 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.990063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.990115 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.990890 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.990891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.990893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:49.990968 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.990970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.991979 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.991980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.993270 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.993273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.994005 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.994006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.994088 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.994092 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.994145 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.994188 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 15:26:49.994951 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.990206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:46423 (local address 127.1.163.66:40009)
0925 15:26:49.990252 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.990254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.990261 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:26:49.990291 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.990292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.990301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.990348 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:49.991056 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.991848 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.993383 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.993908 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.994010 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.994020 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.994819 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.994868 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:26:49.995751 13198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c76549f74192964d06e53d5b6693" candidate_uuid: "29808b0f05014c96b656191844acda57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" is_pre_election: true
I20260925 15:26:49.995937 13198 raft_consensus.cc:2454] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29808b0f05014c96b656191844acda57 in term 0.
I20260925 15:26:49.996194 13268 leader_election.cc:302] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29808b0f05014c96b656191844acda57, 5036d6ac7ddb4c1cad246a9e9a96a2d1; no voters:
I20260925 15:26:49.996331 13403 raft_consensus.cc:2781] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:26:49.996377 13403 raft_consensus.cc:482] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:26:49.996415 13403 raft_consensus.cc:3037] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:49.997262 13403 raft_consensus.cc:504] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:49.997439 13403 leader_election.cc:288] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [CANDIDATE]: Term 1 election: Requested vote from peers 5036d6ac7ddb4c1cad246a9e9a96a2d1 (127.1.163.66:40009), ab22f721d60c41b69d291d5683812803 (127.1.163.65:41077)
W20260925 15:26:49.997529 13108 tablet.cc:2369] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:26:49.997666 13198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c76549f74192964d06e53d5b6693" candidate_uuid: "29808b0f05014c96b656191844acda57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1"
I20260925 15:26:49.997737 13198 raft_consensus.cc:3037] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:49.998443 13198 raft_consensus.cc:2454] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29808b0f05014c96b656191844acda57 in term 1.
I20260925 15:26:49.998615 13268 leader_election.cc:302] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29808b0f05014c96b656191844acda57, 5036d6ac7ddb4c1cad246a9e9a96a2d1; no voters:
I20260925 15:26:49.999257 13403 raft_consensus.cc:2781] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:26:49.999459 13403 raft_consensus.cc:686] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 1 LEADER]: Becoming Leader. State: Replica: 29808b0f05014c96b656191844acda57, State: Running, Role: LEADER
I20260925 15:26:49.999619 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.990502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:41077 (local address 127.1.163.67:53827)
0925 15:26:49.994373 (+ 3871us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:49.994386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:26:49.994437 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:26:49.994609 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:26:49.994610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:26:49.994611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:26:49.994645 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.994646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.995526 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.995527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.996498 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:26:49.996501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.999216 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.999217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:26:49.999278 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.999282 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:26:49.999386 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:26:49.999435 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3856,"thread_start_us":119,"threads_started":1}
I20260925 15:26:49.999804 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:49.990750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53827 (local address 127.1.163.65:41077)
0925 15:26:49.990800 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:49.990802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:49.994494 (+ 3692us) server_negotiation.cc:407] Connection header received
0925 15:26:49.994526 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:49.994527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:49.994540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:49.994597 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:26:49.994713 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.995425 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.998653 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:49.999108 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:49.999217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:49.999228 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:49.999680 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 15:26:49.999739 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:26:49.999567 13403 consensus_queue.cc:237] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } }
I20260925 15:26:50.000072 13061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c76549f74192964d06e53d5b6693" candidate_uuid: "29808b0f05014c96b656191844acda57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab22f721d60c41b69d291d5683812803" is_pre_election: true
I20260925 15:26:50.000218 13061 raft_consensus.cc:2454] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29808b0f05014c96b656191844acda57 in term 0.
I20260925 15:26:50.000326 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7285c76549f74192964d06e53d5b6693" candidate_uuid: "29808b0f05014c96b656191844acda57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab22f721d60c41b69d291d5683812803"
I20260925 15:26:50.000376 13060 raft_consensus.cc:3037] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:26:50.001000 12894 catalog_manager.cc:5495] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29808b0f05014c96b656191844acda57 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "29808b0f05014c96b656191844acda57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29808b0f05014c96b656191844acda57" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 35217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 } health_report { overall_health: UNKNOWN } } }
I20260925 15:26:50.001341 13060 raft_consensus.cc:2454] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29808b0f05014c96b656191844acda57 in term 1.
I20260925 15:26:50.026147 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:50.022877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.1.163.67:35217)
0925 15:26:50.022926 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:26:50.022928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:26:50.022943 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:26:50.023042 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:26:50.023043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:26:50.023065 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:26:50.023143 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:26:50.023281 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:50.024019 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:50.025155 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:26:50.025482 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:26:50.025563 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:26:50.025723 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:26:50.025884 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:26:50.026000 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:26:50.026019 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:26:50.026044 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:26:50.027464 13246 tablet.cc:2369] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:26:50.028288 13198 raft_consensus.cc:1261] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1 [term 1 FOLLOWER]: Refusing update from remote peer 29808b0f05014c96b656191844acda57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:26:50.028295 13060 raft_consensus.cc:1261] T 7285c76549f74192964d06e53d5b6693 P ab22f721d60c41b69d291d5683812803 [term 1 FOLLOWER]: Refusing update from remote peer 29808b0f05014c96b656191844acda57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:26:50.028659 13403 consensus_queue.cc:1035] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5036d6ac7ddb4c1cad246a9e9a96a2d1" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:26:50.028815 13408 consensus_queue.cc:1035] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab22f721d60c41b69d291d5683812803" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:26:50.032387 13411 mvcc.cc:205] Tried to move back new op lower bound from 7333273641072848896 to 7333273640960675840. Current Snapshot: MvccSnapshot[applied={T|T < 7333273641072848896}]
I20260925 15:26:50.032866 13412 mvcc.cc:205] Tried to move back new op lower bound from 7333273641072848896 to 7333273640960675840. Current Snapshot: MvccSnapshot[applied={T|T < 7333273641072848896}]
I20260925 15:26:50.034379 13413 mvcc.cc:205] Tried to move back new op lower bound from 7333273641072848896 to 7333273640960675840. Current Snapshot: MvccSnapshot[applied={T|T < 7333273641072848896}]
I20260925 15:26:50.036208 12895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:26:50.036331 12895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:26:50.037106 12895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea623bc943b54e95bafe53601ffaa38b: Sending DeleteTablet for 3 replicas of tablet 7285c76549f74192964d06e53d5b6693
I20260925 15:26:50.037462 13041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7285c76549f74192964d06e53d5b6693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:58002
I20260925 15:26:50.037866 13179 tablet_service.cc:1514] Processing DeleteTablet for tablet 7285c76549f74192964d06e53d5b6693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:32882
I20260925 15:26:50.038198 13313 tablet_service.cc:1514] Processing DeleteTablet for tablet 7285c76549f74192964d06e53d5b6693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:26:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:58710
I20260925 15:26:50.038713 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 12976
I20260925 15:26:50.041972 13422 tablet_replica.cc:330] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: stopping tablet replica
I20260925 15:26:50.042098 13422 raft_consensus.cc:2229] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:26:50.042232 13422 raft_consensus.cc:2258] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:26:50.042737 13422 ts_tablet_manager.cc:1905] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:26:50.046468 12882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:41077 (error 108)
I20260925 15:26:50.047389 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:50.047249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:41077 (local address 127.0.0.1:58010)
0925 15:26:50.047314 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:50.047351 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:41077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:26:50.047638 12882 catalog_manager.cc:4584] TS ab22f721d60c41b69d291d5683812803 (127.1.163.65:41077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7285c76549f74192964d06e53d5b6693: Network error: Client connection negotiation failed: client connection to 127.1.163.65:41077: connect: Connection refused (error 111)
I20260925 15:26:50.047899 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 13113
I20260925 15:26:50.047960 13421 tablet_replica.cc:330] T 7285c76549f74192964d06e53d5b6693 P 5036d6ac7ddb4c1cad246a9e9a96a2d1: stopping tablet replica
I20260925 15:26:50.048669 13422 ts_tablet_manager.cc:1918] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:26:50.048746 13422 log.cc:1196] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/7285c76549f74192964d06e53d5b6693
I20260925 15:26:50.049038 13422 ts_tablet_manager.cc:1939] T 7285c76549f74192964d06e53d5b6693 P 29808b0f05014c96b656191844acda57: Deleting consensus metadata
I20260925 15:26:50.052611 12881 catalog_manager.cc:4841] TS 29808b0f05014c96b656191844acda57 (127.1.163.67:35217): tablet 7285c76549f74192964d06e53d5b6693 (table test-table [id=48ee271e5a0d41f081aef75c47eda5e5]) successfully deleted
I20260925 15:26:50.055442 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 13250
I20260925 15:26:50.056511 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:26:50.056384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:40009 (local address 127.0.0.1:32890)
0925 15:26:50.056452 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:26:50.056478 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.66:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:26:50.060794 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 12863
I20260925 15:26:50.069528 1677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:26:50.408290 1677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:26:50.408440 1677 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:26:50.753875 1677 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:26:50.754 UTC [12449] LOG: received smart shutdown request
2026-09-25 15:26:50.759 UTC [12449] LOG: background worker "logical replication launcher" (PID 12458) exited with exit code 1
2026-09-25 15:26:50.759 UTC [12453] LOG: shutting down
2026-09-25 15:26:50.982 UTC [12449] LOG: database system is shut down
2026-09-25T15:26:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48250 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:26:51 dist-test-slave-hvv5 krb5kdc[13432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:26:51 dist-test-slave-hvv5 krb5kdc[13432](info): set up 2 sockets
Sep 25 15:26:51 dist-test-slave-hvv5 krb5kdc[13432](info): commencing operation
krb5kdc: starting...
W20260925 15:26:52.140723 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:26:52 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:26:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangeradmin_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangeradmin_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangerlookup_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangerlookup_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
I20260925 15:26:52.332403 1677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk86Bl6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/postgres -l logfile start
2026-09-25 15:26:55.603 UTC [13462] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:26:55.603 UTC [13462] LOG: listening on IPv4 address "127.1.163.84", port 43231
2026-09-25 15:26:55.608 UTC [13462] LOG: listening on Unix socket "/tmp/.s.PGSQL.43231"
2026-09-25 15:26:55.631 UTC [13465] LOG: database system was shut down at 2026-09-25 15:26:53 UTC
2026-09-25 15:26:55.636 UTC [13462] LOG: database system is ready to accept connections
I20260925 15:26:56.600436 1677 mini_postgres.cc:96] Postgres bound to 43231
2026-09-25 15:26:56.606 UTC [13473] FATAL: database "slave" does not exist
127.1.163.84:43231 - accepting connections
I20260925 15:26:56.607456 1677 mini_ranger.cc:161] Starting Ranger...
I20260925 15:26:56.623140 1677 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:26:57.277525 1677 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:26:57.277612 1677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin
2026-09-25 15:26:57,558 [I] DB FLAVOR :POSTGRES
2026-09-25 15:26:57,559 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:26:57,559 [I] Checking connection..
2026-09-25 15:26:57,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:26:57,830 [I] Checking connection passed.
2026-09-25 15:26:57,830 [I] --------- Verifying version history table ---------
2026-09-25 15:26:57,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:58,084 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:26:58,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:58,351 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:26:58,351 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:26:58,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:26:58.596 UTC [13537] WARNING: there is no transaction in progress
2026-09-25 15:26:58,603 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:26:58,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:26:58,856 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:26:58,856 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:26:58,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:26:59,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:26:59,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvv5.c.gcp-upstream.internal','N') ;"
2026-09-25 15:26:59,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:26:59,870 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:26:59,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:27:00,109 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:27:00,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:27:00,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:27:00,644 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:27:00,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:27:01.746 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.766 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.786 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.804 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.824 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.844 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.905 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.913 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.929 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.941 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.953 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.969 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.980 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.988 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:01.996 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:02.006 UTC [13645] WARNING: there is no transaction in progress
2026-09-25 15:27:02,788 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:27:02,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:27:03,069 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:27:03,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:27:03,326 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:27:03,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:27:03,602 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:27:03,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:27:03,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:27:04,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvv5.c.gcp-upstream.internal';"
2026-09-25 15:27:04,441 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:27:04,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:27:04,715 [I] DB_PATCHES have already been applied
I20260925 15:27:04.721993 1677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk86Bl6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:27:04.722070 1677 mini_ranger.cc:193] Using host: 127.1.163.84
I20260925 15:27:04.725482 1677 mini_ranger.cc:227] Ranger admin URL: http://127.1.163.84:50125
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:27:05.881620 1677 mini_ranger.cc:160] Time spent starting Ranger: real 9.274s user 0.000s sys 0.009s
Sep 25, 2026 3:27:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:27:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:27:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:27:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:27:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:27:29.122006 1677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.163.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.163.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangerkms_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.163.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/rangerkms_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.163.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.163.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.163.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/HTTP_127.1.163.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:27:29 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350049, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:27:29.212744 1677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:27:29.223089 1677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:27:30.301090 1677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:27:30.301182 1677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms
I20260925 15:27:30.301204 1677 mini_ranger_kms.cc:227] Using postgres at 127.1.163.84:43231
2026-09-25 15:27:30,472 [I] DB FLAVOR :POSTGRES
2026-09-25 15:27:30,472 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:27:30,472 [I] Checking connection
2026-09-25 15:27:30,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:27:30,720 [I] connection success
2026-09-25 15:27:30,720 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:27:30,720 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:27:30,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:27:30,963 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:27:30,964 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:27:30,964 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:27:30,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.163.84:43231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:27:31,248 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:27:31.807741 1677 mini_ranger_kms.cc:310] Created kms service
I20260925 15:27:31.980265 1677 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:27:32.027627 1677 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:27:32 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350049, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:32.451623 1677 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:27:32.451753 1677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk86Bl6/build/release/bin/postgresql.jar:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk86Bl6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:27:32.451774 1677 mini_ranger_kms.cc:240] Using host: 127.1.163.84
I20260925 15:27:32.455277 1677 mini_ranger_kms.cc:276] Ranger KMS PID: 13826
I20260925 15:27:32.455369 1677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.163.84:37359
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:27:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:27:34.735276 1677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.523s user 0.001s sys 0.153s
I20260925 15:27:34.735396 1677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:27:34.735458 1677 mini_ranger_kms.cc:401] 127.1.163.84:37359/kms/v1/keys
Sep 25 15:27:34 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790350054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:27:35 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:27:35 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.163.84@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:27:37 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:27:38.017510 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34885
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:34885
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:38.103375 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:38.103634 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:38.103672 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:38.104928 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:38.104996 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:38.105023 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:38.105046 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:38.105077 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:38.106735 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.163.84:37359/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:34885
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:38.107033 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:38.107309 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:38.113523 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:38.113543 13876 server_base.cc:1034] running on GCE node
W20260925 15:27:38.113649 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:38.113517 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:38.114063 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:38.114480 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:38.115658 13876 hybrid_clock.cc:648] HybridClock initialized: now 1790350058115644 us; error 30 us; skew 500 ppm
Sep 25 15:27:38 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:38.126215 13876 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:38.126942 13876 webserver.cc:460] Webserver started at http://127.1.163.126:36779/ using document root <none> and password file <none>
I20260925 15:27:38.127189 13876 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:38.127251 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:38.127413 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:27:38 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:38.189782 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "04fabd73d67e4ac695b10aacf665b84d"
format_stamp: "Formatted at 2026-09-25 15:27:38 on dist-test-slave-hvv5"
server_key: "54434eeb5b997df3f4ad2d0f61bea81a"
server_key_iv: "299e3752aabcf600b7332f87f2f3c853"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:38.190279 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "04fabd73d67e4ac695b10aacf665b84d"
format_stamp: "Formatted at 2026-09-25 15:27:38 on dist-test-slave-hvv5"
server_key: "54434eeb5b997df3f4ad2d0f61bea81a"
server_key_iv: "299e3752aabcf600b7332f87f2f3c853"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:38.191907 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:38.204605 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:38.204808 13876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:38.204936 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "04fabd73d67e4ac695b10aacf665b84d"
format_stamp: "Formatted at 2026-09-25 15:27:38 on dist-test-slave-hvv5"
server_key: "54434eeb5b997df3f4ad2d0f61bea81a"
server_key_iv: "299e3752aabcf600b7332f87f2f3c853"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:38.205034 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:38.217352 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:38.219847 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:38.220013 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:38.225399 13876 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:34885
I20260925 15:27:38.225495 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:34885 every 8 connection(s)
I20260925 15:27:38.225894 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:38.226686 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:38.229323 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Bootstrap starting.
I20260925 15:27:38.229964 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:38.230268 13944 log.cc:824] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:38.230993 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: No bootstrap required, opened a new log
I20260925 15:27:38.233186 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:38.233367 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:38.233429 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fabd73d67e4ac695b10aacf665b84d, State: Initialized, Role: FOLLOWER
I20260925 15:27:38.233595 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:38.233677 13944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:38.233721 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:38.233769 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:38.234331 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:38.234481 13944 leader_election.cc:302] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fabd73d67e4ac695b10aacf665b84d; no voters:
I20260925 15:27:38.234603 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 13876
I20260925 15:27:38.234812 13944 leader_election.cc:288] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:27:38.234850 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:27:38.234863 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:38.235020 13947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 LEADER]: Becoming Leader. State: Replica: 04fabd73d67e4ac695b10aacf665b84d, State: Running, Role: LEADER
I20260925 15:27:38.235065 1677 external_mini_cluster.cc:1428] Setting key 7e6964c171b357d9de8707254b948230
I20260925 15:27:38.235102 13947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:38.235146 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:38.235404 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04fabd73d67e4ac695b10aacf665b84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } } }
I20260925 15:27:38.235466 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:38.235565 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fabd73d67e4ac695b10aacf665b84d. Latest consensus state: current_term: 1 leader_uuid: "04fabd73d67e4ac695b10aacf665b84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } } }
I20260925 15:27:38.235600 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:38.236791 13944 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:27:38.239879 13951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:38.241122 13951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:38.243992 13951 catalog_manager.cc:1347] Generated new cluster ID: 3b1da4f036b848238b295aa864f10d6d
I20260925 15:27:38.244055 13951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:27:38 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:38.251190 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:38.235793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.1.163.126:34885)
0925 15:27:38.235963 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:38.235974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:38.236274 (+ 300us) server_negotiation.cc:407] Connection header received
0925 15:27:38.236456 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:38.236464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:38.236525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:38.236715 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:38.239045 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:38.242447 (+ 3402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:38.243665 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:38.244325 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:38.244441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:38.247313 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:38.247317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:38.247317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:38.247331 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:38.249345 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:38.249738 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:38.249741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:38.249742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:38.249834 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:38.250001 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:38.250002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:38.250002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:38.250295 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:38.250428 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:38.250834 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:27:38.250958 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260925 15:27:38.253929 13951 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:27:38.254454 13951 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:38.259119 13951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Generated new TSK 0
I20260925 15:27:38.259308 13951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:27:38.981101 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:39.073323 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:39.073550 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:39.073575 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:39.074770 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:39.074819 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:39.074846 13989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:27:39.076336 13989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:39.076633 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:39.076884 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:39.082130 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.082113 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.082108 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:39.082783 13989 server_base.cc:1034] running on GCE node
I20260925 15:27:39.083014 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:39.083256 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:39.084424 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790350059084409 us; error 33 us; skew 500 ppm
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:39.087131 13989 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:27:39.087646 13989 webserver.cc:460] Webserver started at http://127.1.163.65:41341/ using document root <none> and password file <none>
I20260925 15:27:39.087890 13989 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:39.087968 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:39.088120 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:39.111174 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "fd9f012c9b6649fc84962153160765fa"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "23a876afa6e901105567fec1e62dc551"
server_key_iv: "df31fcbef6a6f38e94d0a58284abf293"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.111589 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "fd9f012c9b6649fc84962153160765fa"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "23a876afa6e901105567fec1e62dc551"
server_key_iv: "df31fcbef6a6f38e94d0a58284abf293"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.113178 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:27:39.134449 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.134696 13989 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 15:27:39.134903 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "fd9f012c9b6649fc84962153160765fa"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "23a876afa6e901105567fec1e62dc551"
server_key_iv: "df31fcbef6a6f38e94d0a58284abf293"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.135008 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:39.144387 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:39.146688 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:39.146875 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:39.147154 13989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:39.147601 13989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:39.147662 13989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.147706 13989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:39.147733 13989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.155009 13989 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:45485
I20260925 15:27:39.155476 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:27:39.160774 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 13989
I20260925 15:27:39.160885 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:27:39.161085 1677 external_mini_cluster.cc:1428] Setting key 09825c858cc32b3a7f4dd4ebcc07ef7b
I20260925 15:27:39.156212 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:45485 every 8 connection(s)
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:39.168840 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.157412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:43673 (local address 127.1.163.126:34885)
0925 15:27:39.157567 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:39.157569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:39.157581 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:27:39.158093 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:39.158094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:39.158110 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:39.158158 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:39.159103 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.159731 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.161517 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.161841 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.161889 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.164789 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:39.164792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:39.164792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:39.164801 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:39.166884 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.167509 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.167511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.167511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.167587 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.167931 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.167932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.167932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.168093 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:39.168176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:39.168607 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 15:27:39.168713 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260925 15:27:39.169221 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.156981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.65:43673)
0925 15:27:39.157192 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:39.157252 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:39.157957 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:39.158364 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:39.158369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:39.158651 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:39.158978 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.158985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.159839 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.159839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.161425 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.161428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.162001 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.162002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.162153 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.162828 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:39.162839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:39.164657 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:39.166996 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.167007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.167011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.167413 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.167672 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.167673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.167674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.167842 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.168181 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:39.168189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:39.168414 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 15:27:39.168585 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 15:27:39.168934 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260925 15:27:39.169878 14119 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:39.169996 14119 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:39.170209 14119 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:39.170933 13908 ts_manager.cc:194] Registered new tserver with Master: fd9f012c9b6649fc84962153160765fa (127.1.163.65:45485)
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:27:39.171883 13908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:43673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:27:39.192987 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:39.315111 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:39.315478 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:39.315537 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:39.317423 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:39.317484 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:39.317528 14126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:27:39.319921 14126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:39.320204 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:39.320519 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:39.327268 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.327600 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.327732 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:39.327973 14126 server_base.cc:1034] running on GCE node
I20260925 15:27:39.328303 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:39.328660 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:39.329842 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790350059329819 us; error 38 us; skew 500 ppm
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:39.333303 14126 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:27:39.333910 14126 webserver.cc:460] Webserver started at http://127.1.163.66:34897/ using document root <none> and password file <none>
I20260925 15:27:39.334193 14126 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:39.334263 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:39.334453 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:39.353602 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "3462599134a0411780bcbc6984ccb028"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "dd4b3f500f96573bb49c88da4b10efc1"
server_key_iv: "8f315b080aa147429fc4f9dd02167500"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.354121 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "3462599134a0411780bcbc6984ccb028"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "dd4b3f500f96573bb49c88da4b10efc1"
server_key_iv: "8f315b080aa147429fc4f9dd02167500"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.356120 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:27:39.384655 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.384950 14126 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260925 15:27:39.385054 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "3462599134a0411780bcbc6984ccb028"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "dd4b3f500f96573bb49c88da4b10efc1"
server_key_iv: "8f315b080aa147429fc4f9dd02167500"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.385162 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:39.399376 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:39.402083 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:39.402303 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:39.402585 14126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:39.403046 14126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:39.403183 14126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.403239 14126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:39.403303 14126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.411288 14126 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:46293
I20260925 15:27:39.411448 14256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:46293 every 8 connection(s)
I20260925 15:27:39.411857 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:39.420791 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14126
I20260925 15:27:39.420914 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:27:39.421128 1677 external_mini_cluster.cc:1428] Setting key f761157a25bc7d119eb6a2f0613ac5eb
I20260925 15:27:39.424470 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.413010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:42687 (local address 127.1.163.126:34885)
0925 15:27:39.413057 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:39.413059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:39.413071 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:27:39.413775 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:39.413776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:39.413794 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:39.413827 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:39.414718 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.415504 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.417110 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.417428 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.417515 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.420235 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:39.420237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:39.420238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:39.420247 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:39.422739 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.423278 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.423279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.423280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.423346 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.423663 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.423665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.423665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.423811 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:39.423854 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:39.424255 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 15:27:39.424356 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:27:39.425091 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.412721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.66:42687)
0925 15:27:39.412917 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:39.412968 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:39.413633 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:39.414026 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:39.414030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:39.414287 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:39.414591 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.414598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.415637 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.415638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.416995 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.416999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.417548 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.417549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.417695 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.418375 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:39.418386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:39.420093 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:39.422849 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.422858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.422861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.423182 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.423433 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.423433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.423434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.423578 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.423901 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:39.423908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:39.424088 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:27:39.424642 (+ 554us) client_negotiation.cc:240] Negotiation successful
0925 15:27:39.424837 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260925 15:27:39.425772 14257 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:39.425872 14257 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:39.426076 14257 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:39.426565 13908 ts_manager.cc:194] Registered new tserver with Master: 3462599134a0411780bcbc6984ccb028 (127.1.163.66:46293)
I20260925 15:27:39.427070 13908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:42687
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:27:39.460405 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:39.589651 14263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:39.590044 14263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:39.590137 14263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:39.592223 14263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:39.592626 14263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:39.592814 14263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:27:39.595372 14263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:39.595906 14263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:39.596259 14263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:39.602833 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.602833 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.603132 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:39.604187 14263 server_base.cc:1034] running on GCE node
I20260925 15:27:39.604502 14263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:39.604832 14263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:39.606017 14263 hybrid_clock.cc:648] HybridClock initialized: now 1790350059605996 us; error 35 us; skew 500 ppm
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:39.609872 14263 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:27:39.610410 14263 webserver.cc:460] Webserver started at http://127.1.163.67:45535/ using document root <none> and password file <none>
I20260925 15:27:39.610666 14263 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:39.610714 14263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:39.610847 14263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:39.630050 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "2a27b4395c2d4c03bf734858aa3249a1"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "0658fd408a8f51f0b11c38eeab153138"
server_key_iv: "480bb2cae465779d69763f012dc61033"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.630635 14263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "2a27b4395c2d4c03bf734858aa3249a1"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "0658fd408a8f51f0b11c38eeab153138"
server_key_iv: "480bb2cae465779d69763f012dc61033"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.632912 14263 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:27:39.655445 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.655695 14263 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 15:27:39.655807 14263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "2a27b4395c2d4c03bf734858aa3249a1"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "0658fd408a8f51f0b11c38eeab153138"
server_key_iv: "480bb2cae465779d69763f012dc61033"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.655921 14263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:39.669603 14263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:39.673966 14263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:39.674204 14263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:39.674527 14263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:39.674937 14263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:39.674994 14263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.675063 14263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:39.675092 14263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.682595 14263 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:41879
I20260925 15:27:39.683017 14263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:27:39.684837 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:41879 every 8 connection(s)
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:39.691828 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14263
I20260925 15:27:39.691946 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:27:39.692162 1677 external_mini_cluster.cc:1428] Setting key 2c72d76aa0a57bda9b3612c4813f1b12
I20260925 15:27:39.693881 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.683824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:43061 (local address 127.1.163.126:34885)
0925 15:27:39.683903 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:39.683906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:39.684166 (+ 260us) server_negotiation.cc:407] Connection header received
0925 15:27:39.684842 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:39.684843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:39.684857 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:39.684891 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:39.685626 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.686155 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.687245 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.687486 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.687576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.690148 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:39.690151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:39.690151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:39.690167 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:39.692177 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.692767 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.692768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.692769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.692836 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:39.693160 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:39.693161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:39.693161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:39.693294 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:39.693334 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:39.693695 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 15:27:39.693803 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:27:39.694408 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:39.683848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.67:43061)
0925 15:27:39.684009 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:39.684066 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:39.684697 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:39.685096 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:39.685101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:39.685328 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:39.685526 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.685530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.686272 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.686273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.687131 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:39.687134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:39.687599 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:39.687600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:39.687927 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:39.688617 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:39.688626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:39.690033 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:39.692287 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.692296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.692299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.692667 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.692912 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:39.692913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:39.692913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:39.693080 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:39.693363 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:39.693367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:39.693527 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 15:27:39.693993 (+ 466us) client_negotiation.cc:240] Negotiation successful
0925 15:27:39.694177 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260925 15:27:39.695039 14392 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:39.695166 14392 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:39.695365 14392 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:39.695767 13908 ts_manager.cc:194] Registered new tserver with Master: 2a27b4395c2d4c03bf734858aa3249a1 (127.1.163.67:41879)
I20260925 15:27:39.696161 13908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:43061
I20260925 15:27:39.704152 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:27:39.704939 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 13989
I20260925 15:27:39.718842 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14126
I20260925 15:27:39.725283 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14263
I20260925 15:27:39.732730 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 13876
I20260925 15:27:39.738332 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34885
--webserver_interface=127.1.163.126
--webserver_port=36779
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:34885
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:39.824801 14395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:39.825065 14395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:39.825106 14395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:39.826371 14395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:39.826478 14395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:39.826509 14395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:39.826539 14395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:39.826565 14395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:39.828277 14395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:34885
--ranger_config_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:39.828621 14395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:39.828874 14395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:39.834118 14400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:39.834180 14403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:39.834380 14395 server_base.cc:1034] running on GCE node
W20260925 15:27:39.834507 14401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:39.834798 14395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:39.835173 14395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:39.836340 14395 hybrid_clock.cc:648] HybridClock initialized: now 1790350059836320 us; error 40 us; skew 500 ppm
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:39.838984 14395 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:39.839525 14395 webserver.cc:460] Webserver started at http://127.1.163.126:36779/ using document root <none> and password file <none>
I20260925 15:27:39.839763 14395 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:39.839813 14395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:27:39 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:39.901130 14395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:39.901932 14411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:39.902153 14395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:39.902280 14395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "04fabd73d67e4ac695b10aacf665b84d"
format_stamp: "Formatted at 2026-09-25 15:27:38 on dist-test-slave-hvv5"
server_key: "54434eeb5b997df3f4ad2d0f61bea81a"
server_key_iv: "299e3752aabcf600b7332f87f2f3c853"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:39.902783 14395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:39.924659 14395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:39.926894 14395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:39.927021 14395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:39.931607 14395 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:34885
I20260925 15:27:39.931658 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:34885 every 8 connection(s)
I20260925 15:27:39.932050 14395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:39.933429 14464 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:27:39.934024 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Bootstrap starting.
I20260925 15:27:39.934649 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14395
I20260925 15:27:39.934998 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:45485
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:34885
--webserver_port=41341
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:39.944593 14464 log.cc:824] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:39.947192 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:27:39.947575 14464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Bootstrap complete.
I20260925 15:27:39.951020 14464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:39.951370 14464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fabd73d67e4ac695b10aacf665b84d, State: Initialized, Role: FOLLOWER
I20260925 15:27:39.951541 14464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:39.951602 14464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:39.951650 14464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:39.951704 14464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:27:39.953208 14464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:39.953341 14464 leader_election.cc:302] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fabd73d67e4ac695b10aacf665b84d; no voters:
I20260925 15:27:39.953511 14464 leader_election.cc:288] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:27:39.953614 14468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:27:39.953775 14464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:39.953821 14468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [term 2 LEADER]: Becoming Leader. State: Replica: 04fabd73d67e4ac695b10aacf665b84d, State: Running, Role: LEADER
I20260925 15:27:39.954110 14468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } }
I20260925 15:27:39.954489 14469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04fabd73d67e4ac695b10aacf665b84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } } }
I20260925 15:27:39.954583 14469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:39.954815 14470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fabd73d67e4ac695b10aacf665b84d. Latest consensus state: current_term: 2 leader_uuid: "04fabd73d67e4ac695b10aacf665b84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fabd73d67e4ac695b10aacf665b84d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34885 } } }
I20260925 15:27:39.954877 14470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:39.955017 14477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:39.955374 14464 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:27:39.955603 14477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:39.955788 14477 catalog_manager.cc:1259] Loaded cluster ID: 3b1da4f036b848238b295aa864f10d6d
I20260925 15:27:39.955852 14477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:27:39.960165 14477 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:39.960480 14477 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Loaded TSK: 0
I20260925 15:27:39.960794 14477 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:27:40.060477 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:40.060931 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:40.061022 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:40.063007 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:40.063151 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:40.063241 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:27:40.065713 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=41341
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:40.066251 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:40.066614 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:40.073249 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:40.073271 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:40.073575 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:40.073845 14466 server_base.cc:1034] running on GCE node
I20260925 15:27:40.074096 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:40.074416 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:40.075654 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790350060075597 us; error 74 us; skew 500 ppm
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:40.079267 14466 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:27:40.079876 14466 webserver.cc:460] Webserver started at http://127.1.163.65:41341/ using document root <none> and password file <none>
I20260925 15:27:40.080193 14466 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:40.080287 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:40.100364 14466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:40.102644 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.102960 14466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 15:27:40.103102 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "fd9f012c9b6649fc84962153160765fa"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "23a876afa6e901105567fec1e62dc551"
server_key_iv: "df31fcbef6a6f38e94d0a58284abf293"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:40.103595 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:40.126341 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:40.128657 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:40.128831 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:40.129124 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:40.129519 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:40.129575 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.129638 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:40.129671 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.137154 14466 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:45485
I20260925 15:27:40.137244 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:45485 every 8 connection(s)
I20260925 15:27:40.137636 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:27:40.145881 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14466
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:40.146193 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:46293
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:34885
--webserver_port=34897
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:40.150995 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.138449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.65:55291)
0925 15:27:40.138647 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:40.138700 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:40.139364 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:40.140137 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:40.140148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:40.140363 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:40.140721 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.140728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.142453 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.142454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.143258 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.143261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.144132 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.144133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.144235 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.144913 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:40.144921 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:40.146511 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:40.148892 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.148906 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.148909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.149235 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.149563 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.149564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.149564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.149716 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.150245 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:40.150252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:40.150419 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:27:40.150547 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:27:40.150721 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 15:27:40.151050 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.138548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:55291 (local address 127.1.163.126:34885)
0925 15:27:40.139598 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:40.139608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:40.139651 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:27:40.139807 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:40.139813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:40.139873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:40.140039 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:40.140908 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.142334 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.143359 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.144017 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.144136 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.146652 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:40.146657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:40.146658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:40.146672 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:40.148787 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.149337 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.149341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.149341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.149480 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.149799 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.149800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.149801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.150157 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:40.150255 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:40.150627 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 15:27:40.150776 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":72,"threads_started":1}
I20260925 15:27:40.160318 14626 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:40.160449 14626 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.160703 14626 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.161926 14626 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:34885 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:27:40.162025 14626 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.162108 14626 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:40.162562 14626 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.162647 14626 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.163199 14626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:27:40.251654 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:40.252005 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:40.252067 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:40.254046 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:40.254166 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:40.254263 14630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:27:40.256253 14630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:46293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=34897
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:40.256595 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:40.256858 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:40.263422 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:40.263422 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:40.263423 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:40.264211 14630 server_base.cc:1034] running on GCE node
I20260925 15:27:40.264457 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:40.264776 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:40.265971 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790350060265953 us; error 33 us; skew 500 ppm
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:40.269487 14630 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:27:40.270272 14630 webserver.cc:460] Webserver started at http://127.1.163.66:34897/ using document root <none> and password file <none>
I20260925 15:27:40.270607 14630 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:40.270737 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:40.297236 14630 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.000s
I20260925 15:27:40.307401 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.307690 14630 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.000s
I20260925 15:27:40.307788 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "3462599134a0411780bcbc6984ccb028"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "dd4b3f500f96573bb49c88da4b10efc1"
server_key_iv: "8f315b080aa147429fc4f9dd02167500"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:40.308363 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:40.330209 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:40.333036 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:40.333287 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:40.333676 14630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:40.334061 14630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:40.334115 14630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.334158 14630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:40.334184 14630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.342031 14630 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:46293
I20260925 15:27:40.342574 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:27:40.343128 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14630
I20260925 15:27:40.343465 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.163.84:37359/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:41879
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:34885
--webserver_port=45535
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:40.343581 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:46293 every 8 connection(s)
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:40.357604 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.344235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:32965 (local address 127.1.163.126:34885)
0925 15:27:40.344285 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:40.344287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:40.344515 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:27:40.346057 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:40.346059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:40.346072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:40.346104 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:40.346981 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.347903 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.349484 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.349821 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.349891 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.352891 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:40.352893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:40.352893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:40.352903 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:40.355538 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.356202 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.356204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.356205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.356273 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.356671 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.356673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.356673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.356840 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:40.356905 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:40.357347 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 15:27:40.357458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:27:40.358608 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.343413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.66:32965)
0925 15:27:40.344385 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:40.344424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:40.345911 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:40.346298 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:40.346302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:40.346568 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:40.346861 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.346867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.348036 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.348037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.349404 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.349407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.349984 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.349985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.350128 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.350785 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:40.350801 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:40.352759 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:40.355687 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.355696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.355699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.356088 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.356368 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.356369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.356369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.356581 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.356942 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:40.356949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:40.357150 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 15:27:40.358052 (+ 902us) client_negotiation.cc:240] Negotiation successful
0925 15:27:40.358291 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":87,"threads_started":1}
I20260925 15:27:40.359431 14760 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:40.359540 14760 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.359759 14760 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.360476 14760 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:34885 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:27:40.360571 14760 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.360632 14760 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:40.360960 14760 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.361027 14760 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.361320 14760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:27:40.461673 14763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:40.461997 14763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:40.462037 14763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:40.463927 14763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:40.463994 14763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:40.464036 14763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:27:40.466403 14763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.163.84:37359/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:41879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=45535
--tserver_master_addrs=127.1.163.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:40.466961 14763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:40.467285 14763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:40.474575 14770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:40.474895 14769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:40.475068 14763 server_base.cc:1034] running on GCE node
W20260925 15:27:40.480106 14772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:40.480559 14763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:40.480927 14763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:40.482137 14763 hybrid_clock.cc:648] HybridClock initialized: now 1790350060482106 us; error 49 us; skew 500 ppm
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:40.486178 14763 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:27:40.486841 14763 webserver.cc:460] Webserver started at http://127.1.163.67:45535/ using document root <none> and password file <none>
I20260925 15:27:40.487174 14763 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:40.487272 14763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:40.505700 14763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:27:40.508183 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.508435 14763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 15:27:40.508608 14763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "2a27b4395c2d4c03bf734858aa3249a1"
format_stamp: "Formatted at 2026-09-25 15:27:39 on dist-test-slave-hvv5"
server_key: "0658fd408a8f51f0b11c38eeab153138"
server_key_iv: "480bb2cae465779d69763f012dc61033"
server_key_version: "kuduclusterkey@0"
I20260925 15:27:40.509141 14763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:40.535365 14763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:40.537995 14763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:40.538251 14763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:40.538556 14763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:40.538954 14763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:40.539031 14763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.539090 14763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:40.539120 14763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:40.546813 14763 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:41879
I20260925 15:27:40.546911 14893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:41879 every 8 connection(s)
I20260925 15:27:40.547449 14763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:27:40.554139 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14763
Sep 25 15:27:40 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:40.558530 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.548260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:57055 (local address 127.1.163.126:34885)
0925 15:27:40.548329 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:40.548332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:40.548527 (+ 195us) server_negotiation.cc:407] Connection header received
0925 15:27:40.549247 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:40.549249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:40.549262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:40.549351 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:40.549929 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.550669 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.552227 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.552495 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.552657 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.555278 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:40.555279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:40.555280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:40.555286 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:40.557020 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.557476 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.557477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.557477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.557537 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.557820 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.557821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.557821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.557944 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:40.558027 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:40.558344 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:27:40.558431 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:27:40.558800 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.548230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34885 (local address 127.1.163.67:57055)
0925 15:27:40.548387 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:40.548431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:40.549126 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:40.549415 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:40.549418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:40.549622 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:40.549826 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.549831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.550797 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.550798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.552111 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:40.552114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.552694 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.552696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:40.552860 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.553506 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:40.553516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:40.555169 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:40.557102 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.557113 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.557121 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.557387 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.557615 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:40.557616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:40.557616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:40.557747 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:40.558017 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:40.558023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:40.558184 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:27:40.558320 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:27:40.558489 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 15:27:40.559466 14894 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34885
I20260925 15:27:40.559574 14894 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.559772 14894 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.560497 14894 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:34885 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:27:40.560607 14894 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.560676 14894 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:40.561079 14894 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:40.561154 14894 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
W20260925 15:27:40.561494 14894 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:27:40.564826 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:40.554598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.1.163.126:34885)
0925 15:27:40.554725 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:40.554727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:40.554956 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:27:40.554981 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:40.554982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:40.554995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:40.555060 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:40.556107 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.556960 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.558647 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:40.559095 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:40.559180 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:40.561747 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:40.561748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:40.561749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:40.561755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:40.563330 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.563699 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.563700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.563701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.563759 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:40.564153 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:40.564155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:40.564155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:40.564320 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:40.564565 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:40.564611 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:27:40.564714 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
Sep 25 15:27:41 dist-test-slave-hvv5 krb5kdc[13432](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790350061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:41.163429 14626 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:41.163625 14626 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:41.164059 14428 ts_manager.cc:194] Registered new tserver with Master: fd9f012c9b6649fc84962153160765fa (127.1.163.65:45485)
I20260925 15:27:41.165150 14428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:55291
I20260925 15:27:41.361495 14760 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:41.361666 14760 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:41.361928 14428 ts_manager.cc:194] Registered new tserver with Master: 3462599134a0411780bcbc6984ccb028 (127.1.163.66:46293)
I20260925 15:27:41.362350 14428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:32965
Sep 25 15:27:41 dist-test-slave-hvv5 krb5kdc[13432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for HTTP/127.1.163.84@KRBTEST.COM
I20260925 15:27:41.561697 14894 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:41.561863 14894 heartbeater.cc:511] Master 127.1.163.126:34885 requested a full tablet report, sending...
I20260925 15:27:41.562263 14427 ts_manager.cc:194] Registered new tserver with Master: 2a27b4395c2d4c03bf734858aa3249a1 (127.1.163.67:41879)
I20260925 15:27:41.562788 14427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:57055
I20260925 15:27:41.573386 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:27:41.583314 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.574487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.1.163.126:34885)
0925 15:27:41.576711 (+ 2224us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.576713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.576722 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:27:41.576747 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.576748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.576758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.576875 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:41.577529 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.578057 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.578838 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.579052 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.579143 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.580513 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:41.580514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:41.580515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:41.580522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:41.582085 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:41.582469 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:41.582471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:41.582471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:41.582535 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:41.582741 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:41.582742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:41.582743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:41.582866 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:41.582974 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.583081 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.583174 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2198,"thread_start_us":69,"threads_started":1}
I20260925 15:27:41.585512 14427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:27:41.587426 14427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:27:41.605003 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:46293 (local address 127.0.0.1:43472)
0925 15:27:41.600704 (+ 3171us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.600742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:41.600801 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:41.601196 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:41.601199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:41.601201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:41.601289 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.601291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.602156 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.602157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.603462 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.603465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.604579 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.604580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.604668 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.604755 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 15:27:41.604833 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:27:41.604898 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3139,"mutex_wait_us":57,"thread_start_us":99,"threads_started":1}
I20260925 15:27:41.605623 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.1.163.66:46293)
0925 15:27:41.601040 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.601044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.601052 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:27:41.601080 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.601082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.601095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.601191 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:41.601389 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.602061 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.603572 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.604327 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.604430 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.604507 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.605494 (+ 987us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.605542 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3190,"thread_start_us":45,"threads_started":1}
I20260925 15:27:41.606575 14680 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 (DEFAULT_TABLE table=test-table [id=eb136016c24142a2b2a0eb0752a4d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:41.607019 14680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa07f2ad11346599726beb8d2410bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:41.607542 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.1.163.65:45485)
0925 15:27:41.602816 (+ 4918us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.602822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.603020 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:27:41.603057 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.603059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.603076 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.603143 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:41.603356 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.604425 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.605812 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.606581 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.606666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.606721 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.607419 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.607463 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4877,"thread_start_us":50,"threads_started":1}
I20260925 15:27:41.607719 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45485 (local address 127.0.0.1:60600)
0925 15:27:41.602884 (+ 5102us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.602895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:41.602963 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:41.603237 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:41.603238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:41.603239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:41.603286 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.603288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.604553 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.604555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.605711 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.605714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.606757 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.606758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.606821 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.606826 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:27:41.606872 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:27:41.606913 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5083,"thread_start_us":63,"threads_started":1}
I20260925 15:27:41.609025 14544 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 (DEFAULT_TABLE table=test-table [id=eb136016c24142a2b2a0eb0752a4d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:41.609340 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa07f2ad11346599726beb8d2410bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:41.612144 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:41879 (local address 127.0.0.1:33546)
0925 15:27:41.600913 (+ 3380us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.600920 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:41.600948 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:41.608310 (+ 7362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:41.608311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:41.608313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:41.608352 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.608353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.609704 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.609705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.610831 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.610833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.611880 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.611881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.611946 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.611950 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:27:41.612024 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:27:41.612065 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3367,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260925 15:27:41.612476 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.597985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.1.163.67:41879)
0925 15:27:41.608063 (+ 10078us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.608068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.608079 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:27:41.608115 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.608117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.608133 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.608211 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:41.608438 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.609609 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.610930 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.611699 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.611773 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.612322 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.612350 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.612401 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10039,"thread_start_us":67,"threads_started":1}
I20260925 15:27:41.613286 14810 tablet_service.cc:1467] Processing CreateTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 (DEFAULT_TABLE table=test-table [id=eb136016c24142a2b2a0eb0752a4d46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:41.613628 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fa07f2ad11346599726beb8d2410bf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:41.617452 14927 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Bootstrap starting.
I20260925 15:27:41.618157 14927 tablet_bootstrap.cc:654] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:41.618548 14927 log.cc:824] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:41.618984 14929 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Bootstrap starting.
I20260925 15:27:41.619678 14929 tablet_bootstrap.cc:654] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:41.620044 14929 log.cc:824] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:41.621029 14927 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: No bootstrap required, opened a new log
I20260925 15:27:41.621104 14927 ts_tablet_manager.cc:1397] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:27:41.623521 14927 raft_consensus.cc:348] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.623693 14927 raft_consensus.cc:374] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:41.623723 14927 raft_consensus.cc:729] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a27b4395c2d4c03bf734858aa3249a1, State: Initialized, Role: FOLLOWER
I20260925 15:27:41.623857 14927 consensus_queue.cc:260] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.624038 14930 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Bootstrap starting.
I20260925 15:27:41.624122 14927 ts_tablet_manager.cc:1428] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:27:41.624495 14894 heartbeater.cc:503] Master 127.1.163.126:34885 was elected leader, sending a full tablet report...
I20260925 15:27:41.624749 14930 tablet_bootstrap.cc:654] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:41.625062 14930 log.cc:824] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:41.626034 14930 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: No bootstrap required, opened a new log
I20260925 15:27:41.626494 14929 tablet_bootstrap.cc:492] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: No bootstrap required, opened a new log
I20260925 15:27:41.626569 14929 ts_tablet_manager.cc:1397] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 15:27:41.626740 14930 ts_tablet_manager.cc:1397] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:27:41.628844 14929 raft_consensus.cc:348] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.629032 14929 raft_consensus.cc:374] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:41.629061 14929 raft_consensus.cc:729] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3462599134a0411780bcbc6984ccb028, State: Initialized, Role: FOLLOWER
I20260925 15:27:41.629298 14929 consensus_queue.cc:260] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.629650 14929 ts_tablet_manager.cc:1428] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:27:41.629660 14930 raft_consensus.cc:348] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.629805 14760 heartbeater.cc:503] Master 127.1.163.126:34885 was elected leader, sending a full tablet report...
I20260925 15:27:41.629906 14930 raft_consensus.cc:374] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:41.630013 14930 raft_consensus.cc:729] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9f012c9b6649fc84962153160765fa, State: Initialized, Role: FOLLOWER
I20260925 15:27:41.630360 14930 consensus_queue.cc:260] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.630776 14626 heartbeater.cc:503] Master 127.1.163.126:34885 was elected leader, sending a full tablet report...
I20260925 15:27:41.630982 14930 ts_tablet_manager.cc:1428] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260925 15:27:41.639216 14627 tablet.cc:2369] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:27:41.650254 14932 raft_consensus.cc:482] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:41.650417 14932 raft_consensus.cc:504] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.650839 14932 leader_election.cc:288] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3462599134a0411780bcbc6984ccb028 (127.1.163.66:46293), fd9f012c9b6649fc84962153160765fa (127.1.163.65:45485)
I20260925 15:27:41.655175 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.651236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:40803 (local address 127.1.163.66:46293)
0925 15:27:41.651294 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.651296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.651458 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:27:41.651565 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.651566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.651576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.651606 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:41.651846 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.652590 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654163 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.654736 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654792 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.654803 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.655047 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.655098 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:27:41.655233 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.651285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34155 (local address 127.1.163.65:45485)
0925 15:27:41.651342 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.651344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.651461 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:27:41.651559 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.651560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.651569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.651598 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:41.651833 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.652631 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654185 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.654774 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654823 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.654832 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.655122 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.655164 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:27:41.655501 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.651087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45485 (local address 127.1.163.67:34155)
0925 15:27:41.651402 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.651412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:41.651486 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:41.651669 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:41.651669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:41.651671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:41.651757 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.651759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.652728 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.652729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.654083 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.654086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.654917 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.654999 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.655003 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:27:41.655340 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 15:27:41.655412 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":49,"thread_start_us":81,"threads_started":1}
I20260925 15:27:41.655894 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa07f2ad11346599726beb8d2410bf2" candidate_uuid: "2a27b4395c2d4c03bf734858aa3249a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9f012c9b6649fc84962153160765fa" is_pre_election: true
I20260925 15:27:41.656087 14580 raft_consensus.cc:2454] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a27b4395c2d4c03bf734858aa3249a1 in term 0.
I20260925 15:27:41.656333 14783 leader_election.cc:302] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a27b4395c2d4c03bf734858aa3249a1, fd9f012c9b6649fc84962153160765fa; no voters:
I20260925 15:27:41.656338 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.651014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:46293 (local address 127.1.163.67:40803)
0925 15:27:41.651393 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.651405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:41.651488 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:41.651672 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:41.651673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:41.651675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:41.651765 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.651766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.652702 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.652703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.654051 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:41.654054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.654884 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.654885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:41.654953 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.654958 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:27:41.655739 (+ 781us) client_negotiation.cc:240] Negotiation successful
0925 15:27:41.656232 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":99,"thread_start_us":110,"threads_started":1}
I20260925 15:27:41.656491 14932 raft_consensus.cc:2781] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:27:41.656550 14932 raft_consensus.cc:482] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:27:41.656584 14932 raft_consensus.cc:3037] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:41.656725 14700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa07f2ad11346599726beb8d2410bf2" candidate_uuid: "2a27b4395c2d4c03bf734858aa3249a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3462599134a0411780bcbc6984ccb028" is_pre_election: true
I20260925 15:27:41.656875 14700 raft_consensus.cc:2454] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a27b4395c2d4c03bf734858aa3249a1 in term 0.
I20260925 15:27:41.657480 14932 raft_consensus.cc:504] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.657646 14932 leader_election.cc:288] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [CANDIDATE]: Term 1 election: Requested vote from peers 3462599134a0411780bcbc6984ccb028 (127.1.163.66:46293), fd9f012c9b6649fc84962153160765fa (127.1.163.65:45485)
I20260925 15:27:41.657828 14700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa07f2ad11346599726beb8d2410bf2" candidate_uuid: "2a27b4395c2d4c03bf734858aa3249a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3462599134a0411780bcbc6984ccb028"
I20260925 15:27:41.657912 14700 raft_consensus.cc:3037] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:41.657927 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fa07f2ad11346599726beb8d2410bf2" candidate_uuid: "2a27b4395c2d4c03bf734858aa3249a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9f012c9b6649fc84962153160765fa"
I20260925 15:27:41.658001 14580 raft_consensus.cc:3037] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:41.658849 14700 raft_consensus.cc:2454] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a27b4395c2d4c03bf734858aa3249a1 in term 1.
I20260925 15:27:41.658955 14580 raft_consensus.cc:2454] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a27b4395c2d4c03bf734858aa3249a1 in term 1.
I20260925 15:27:41.659031 14782 leader_election.cc:302] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a27b4395c2d4c03bf734858aa3249a1, 3462599134a0411780bcbc6984ccb028; no voters:
I20260925 15:27:41.659387 14932 raft_consensus.cc:2781] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:41.659565 14932 raft_consensus.cc:686] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a27b4395c2d4c03bf734858aa3249a1, State: Running, Role: LEADER
I20260925 15:27:41.659664 14932 consensus_queue.cc:237] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } }
I20260925 15:27:41.660509 14427 catalog_manager.cc:5495] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a27b4395c2d4c03bf734858aa3249a1 (127.1.163.67). New cstate: current_term: 1 leader_uuid: "2a27b4395c2d4c03bf734858aa3249a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a27b4395c2d4c03bf734858aa3249a1" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 41879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 } health_report { overall_health: UNKNOWN } } }
I20260925 15:27:41.667488 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.664116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.1.163.67:41879)
0925 15:27:41.664159 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:41.664161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:41.664172 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:27:41.664244 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:41.664244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:41.664263 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:41.664347 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:27:41.664515 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.665352 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.666532 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:41.666814 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:41.666883 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:41.667081 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:27:41.667216 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:27:41.667304 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:41.667390 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:27:41.667411 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:27:41.669698 14580 raft_consensus.cc:1261] T 2fa07f2ad11346599726beb8d2410bf2 P fd9f012c9b6649fc84962153160765fa [term 1 FOLLOWER]: Refusing update from remote peer 2a27b4395c2d4c03bf734858aa3249a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:41.669714 14700 raft_consensus.cc:1261] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 1 FOLLOWER]: Refusing update from remote peer 2a27b4395c2d4c03bf734858aa3249a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:41.670044 14932 consensus_queue.cc:1035] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3462599134a0411780bcbc6984ccb028" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:41.670187 14938 consensus_queue.cc:1035] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9f012c9b6649fc84962153160765fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:41.674474 14941 mvcc.cc:205] Tried to move back new op lower bound from 7333273852595945472 to 7333273852558721024. Current Snapshot: MvccSnapshot[applied={T|T < 7333273852595945472}]
I20260925 15:27:41.674543 14943 mvcc.cc:205] Tried to move back new op lower bound from 7333273852595945472 to 7333273852558721024. Current Snapshot: MvccSnapshot[applied={T|T < 7333273852595945472}]
I20260925 15:27:41.675200 14942 mvcc.cc:205] Tried to move back new op lower bound from 7333273852595945472 to 7333273852558721024. Current Snapshot: MvccSnapshot[applied={T|T < 7333273852595945472}]
I20260925 15:27:41.677881 14427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:41.677990 14427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:41.678923 14427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04fabd73d67e4ac695b10aacf665b84d: Sending DeleteTablet for 3 replicas of tablet 2fa07f2ad11346599726beb8d2410bf2
I20260925 15:27:41.679260 14810 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:41 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:33546
I20260925 15:27:41.679322 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:41 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:60600
I20260925 15:27:41.679716 14680 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fa07f2ad11346599726beb8d2410bf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:41 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:43472
I20260925 15:27:41.680013 14952 tablet_replica.cc:330] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: stopping tablet replica
I20260925 15:27:41.680186 14952 raft_consensus.cc:2229] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:41.680241 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14466
I20260925 15:27:41.683400 14952 raft_consensus.cc:2258] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:41.684056 14952 ts_tablet_manager.cc:1905] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:41.686188 14952 ts_tablet_manager.cc:1918] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:41.686522 14952 log.cc:1196] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/2fa07f2ad11346599726beb8d2410bf2
I20260925 15:27:41.686970 14952 ts_tablet_manager.cc:1939] T 2fa07f2ad11346599726beb8d2410bf2 P 3462599134a0411780bcbc6984ccb028: Deleting consensus metadata
I20260925 15:27:41.687538 14413 catalog_manager.cc:4841] TS 3462599134a0411780bcbc6984ccb028 (127.1.163.66:46293): tablet 2fa07f2ad11346599726beb8d2410bf2 (table test-table [id=eb136016c24142a2b2a0eb0752a4d46d]) successfully deleted
I20260925 15:27:41.692199 14950 tablet_replica.cc:330] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: stopping tablet replica
W20260925 15:27:41.692175 14414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:45485 (error 108)
I20260925 15:27:41.692345 14950 raft_consensus.cc:2229] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:27:41.692505 14950 raft_consensus.cc:2258] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:41.693222 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.693090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:45485 (local address 127.0.0.1:60612)
0925 15:27:41.693145 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.693183 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:27:41.693456 14414 catalog_manager.cc:4584] TS fd9f012c9b6649fc84962153160765fa (127.1.163.65:45485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fa07f2ad11346599726beb8d2410bf2: Network error: Client connection negotiation failed: client connection to 127.1.163.65:45485: connect: Connection refused (error 111)
I20260925 15:27:41.694110 14950 ts_tablet_manager.cc:1905] T 2fa07f2ad11346599726beb8d2410bf2 P 2a27b4395c2d4c03bf734858aa3249a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:41.696034 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14630
I20260925 15:27:41.703128 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14763
I20260925 15:27:41.708899 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:41.708687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:41879 (local address 127.0.0.1:33574)
0925 15:27:41.708812 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:41.708847 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.67:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:27:41.709328 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14395
I20260925 15:27:41.715775 1677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:27:42.061208 1677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:27:42.061517 1677 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:27:42.406601 1677 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:27:42.406 UTC [13462] LOG: received smart shutdown request
2026-09-25 15:27:42.411 UTC [13462] LOG: background worker "logical replication launcher" (PID 13471) exited with exit code 1
2026-09-25 15:27:42.411 UTC [13466] LOG: shutting down
2026-09-25 15:27:42.626 UTC [13462] LOG: database system is shut down
2026-09-25T15:27:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51643 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:27:42 dist-test-slave-hvv5 krb5kdc[14962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:27:42 dist-test-slave-hvv5 krb5kdc[14962](info): set up 2 sockets
Sep 25 15:27:42 dist-test-slave-hvv5 krb5kdc[14962](info): commencing operation
krb5kdc: starting...
W20260925 15:27:43.824671 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:27:43 dist-test-slave-hvv5 krb5kdc[14962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:27:43.951699 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:46157
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:44.037163 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:44.037456 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:44.037513 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:44.038805 14975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:44.038880 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:44.038905 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:44.038939 14975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:44.038965 14975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:44.040693 14975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:46157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:44.040982 14975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:44.041229 14975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:44.046406 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:44.046504 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:44.046452 14975 server_base.cc:1034] running on GCE node
W20260925 15:27:44.046415 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:44.046897 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:44.047192 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:44.048395 14975 hybrid_clock.cc:648] HybridClock initialized: now 1790350064048381 us; error 30 us; skew 500 ppm
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:44.055025 14975 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:44.055505 14975 webserver.cc:460] Webserver started at http://127.1.163.126:46407/ using document root <none> and password file <none>
I20260925 15:27:44.055749 14975 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:44.055820 14975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:44.055981 14975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:44.057199 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "300d77fb441f41aa9514433b9b77ef45"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "7a7f3b49de306c4afeb2125eb0831f4d"
server_key_iv: "3850b9697cad77a2e216f95639bdb927"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.057574 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "300d77fb441f41aa9514433b9b77ef45"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "7a7f3b49de306c4afeb2125eb0831f4d"
server_key_iv: "3850b9697cad77a2e216f95639bdb927"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.059028 14975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:27:44.059860 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:44.060125 14975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:44.060248 14975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "300d77fb441f41aa9514433b9b77ef45"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "7a7f3b49de306c4afeb2125eb0831f4d"
server_key_iv: "3850b9697cad77a2e216f95639bdb927"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.060329 14975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:44.076495 14975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:44.078895 14975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:44.079177 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:44.083573 14975 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:46157
I20260925 15:27:44.083604 15042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:46157 every 8 connection(s)
I20260925 15:27:44.083987 14975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:44.084972 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:44.086622 15043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:27:44.087299 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 14975
I20260925 15:27:44.087404 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:27:44.087622 1677 external_mini_cluster.cc:1428] Setting key 50551163f41a4660d49838749aa93567
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.125@KRBTEST.COM; defaulting to no policy
I20260925 15:27:44.098346 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.087443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:46157 (local address 127.0.0.1:55840)
0925 15:27:44.087666 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.087711 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.088227 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.088519 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.088521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.088951 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.089189 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.089196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.090226 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.090227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.091429 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.091433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.091976 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.091977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.092069 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.092708 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.092720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.094534 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.096553 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.096579 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.096581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.096851 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.097160 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.097162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.097162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.097253 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.097705 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.097711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.097855 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.097934 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.098084 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260925 15:27:44.098349 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.087488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.1.163.126:46157)
0925 15:27:44.087704 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.087716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.087828 (+ 112us) server_negotiation.cc:407] Connection header received
0925 15:27:44.088391 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.088397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.088416 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.088514 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.089311 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.090129 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.091539 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.091868 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.091967 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.094641 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.094648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.094649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.094656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.096393 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.096935 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.096938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.096938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.097023 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.097329 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.097329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.097330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.097583 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.097657 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.098001 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.098167 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":188,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.125@KRBTEST.COM" created.
I20260925 15:27:44.099159 15043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:27:44.099524 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.099411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50834)
0925 15:27:44.099459 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.099503 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:27:44.099697 14991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
I20260925 15:27:44.100293 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.100199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50838)
0925 15:27:44.100247 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.100273 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:27:44.100445 15043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:39167: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
I20260925 15:27:44.116261 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:39167
--webserver_interface=127.1.163.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:44.149762 15043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } attempt: 1
I20260925 15:27:44.150350 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.150181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50846)
0925 15:27:44.150252 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.150304 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:27:44.150810 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.150702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50856)
0925 15:27:44.150760 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.150790 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:27:44.150910 15043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:39167: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
W20260925 15:27:44.202317 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:44.202579 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:44.202602 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:44.203909 15050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:44.203953 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:44.203967 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:44.203980 15050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:44.204028 15050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:44.205626 15050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.163.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:44.205899 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:44.206135 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:44.211144 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:44.211216 15050 server_base.cc:1034] running on GCE node
W20260925 15:27:44.211150 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:44.211150 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:44.211606 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:44.211911 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:44.213085 15050 hybrid_clock.cc:648] HybridClock initialized: now 1790350064213081 us; error 38 us; skew 500 ppm
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:44.215821 15050 init.cc:377] Logged in from keytab as kudu/127.1.163.125@KRBTEST.COM (short username kudu)
I20260925 15:27:44.216342 15050 webserver.cc:460] Webserver started at http://127.1.163.125:33143/ using document root <none> and password file <none>
I20260925 15:27:44.216639 15050 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:44.216717 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:44.216861 15050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:44.217963 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data/instance:
uuid: "24533a49bffe439d9264b9377930b871"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "41bb077f8c4ff8e135398d7d0d72ec27"
server_key_iv: "df81e1cca9dd94a8e3e2fdd95f40cd37"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.218338 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal/instance:
uuid: "24533a49bffe439d9264b9377930b871"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "41bb077f8c4ff8e135398d7d0d72ec27"
server_key_iv: "df81e1cca9dd94a8e3e2fdd95f40cd37"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.219043 15043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } attempt: 2
I20260925 15:27:44.219435 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.219302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50864)
0925 15:27:44.219368 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.219407 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:27:44.219749 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.219659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50872)
0925 15:27:44.219703 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.219726 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:27:44.219800 15050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
W20260925 15:27:44.219880 15043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:39167: Network error: Client connection negotiation failed: client connection to 127.1.163.125:39167: connect: Connection refused (error 111)
I20260925 15:27:44.220634 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:44.220818 15050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:44.220925 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "24533a49bffe439d9264b9377930b871"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "41bb077f8c4ff8e135398d7d0d72ec27"
server_key_iv: "df81e1cca9dd94a8e3e2fdd95f40cd37"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.221019 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:44.242300 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:44.244485 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:44.244714 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:44.248778 15050 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.125:39167
I20260925 15:27:44.248811 15117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.125:39167 every 8 connection(s)
I20260925 15:27:44.249194 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
I20260925 15:27:44.250064 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:44.251539 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15050
I20260925 15:27:44.251662 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-1/wal/instance
I20260925 15:27:44.251857 1677 external_mini_cluster.cc:1428] Setting key 6b912d55a665d2cb1f13a7572758c60d
I20260925 15:27:44.251849 15118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.124@KRBTEST.COM; defaulting to no policy
I20260925 15:27:44.262470 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.252766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.1.163.126:46157)
0925 15:27:44.252844 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.252847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.253124 (+ 277us) server_negotiation.cc:407] Connection header received
0925 15:27:44.253846 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.253847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.253861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.253937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.254680 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.255161 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.256764 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.256988 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.257049 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.259488 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.259489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.259490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.259497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.260964 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.261472 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.261474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.261474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.261539 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.261793 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.261794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.261794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.261917 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.261995 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.262314 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.262386 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:27:44.262653 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.252749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:46157 (local address 127.0.0.1:55852)
0925 15:27:44.252981 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.253023 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.253734 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.254014 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.254018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.254403 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.254601 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.254605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.255256 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.255257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.256679 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.256681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.257083 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.257084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.257252 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.257781 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.257794 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.259391 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.261061 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.261069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.261071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.261379 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.261608 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.261609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.261609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.261723 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.261985 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.261991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.262133 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.262250 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.262398 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.124@KRBTEST.COM" created.
I20260925 15:27:44.263140 15118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:27:44.271406 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.263442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50888)
0925 15:27:44.263493 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.263504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.263547 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.263764 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.263765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.263907 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.263949 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.263950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.265050 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.265051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.265762 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.265765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.266455 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.266455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.266518 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.266962 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.266966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.268218 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.270127 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.270128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.270129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.270349 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.270563 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.270564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.270564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.270629 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.270957 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.270958 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.271011 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.271094 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.271322 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":117}
I20260925 15:27:44.271409 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.263486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.1.163.125:39167)
0925 15:27:44.263626 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.263630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.263650 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:27:44.263675 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.263676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.263687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.263765 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.264069 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.264967 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.265880 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.266378 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.266454 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.268307 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.268313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.268314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.268319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.270034 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.270439 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.270442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.270443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.270494 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.270728 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.270729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.270729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.270878 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.270947 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.271116 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.271241 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260925 15:27:44.272125 15118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:27:44.272485 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.272380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48604)
0925 15:27:44.272425 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.272469 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:27:44.272660 15069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
I20260925 15:27:44.273111 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.272992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48614)
0925 15:27:44.273054 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.273087 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:27:44.273243 15118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:39449: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
I20260925 15:27:44.281278 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:39449
--webserver_interface=127.1.163.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:44.311039 15043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } attempt: 3
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:27:44.318405 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.311448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50896)
0925 15:27:44.311500 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.311515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.311592 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.311751 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.311752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.311954 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.312024 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.312026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.312805 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.312807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.313308 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.313309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.313720 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.313721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.313810 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.314383 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.314388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.315635 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.317329 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.317330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.317330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.317509 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.317729 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.317730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.317730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.317793 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.318059 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.318061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.318107 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.318193 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.318297 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:27:44.318405 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.311515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.1.163.125:39167)
0925 15:27:44.311585 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.311587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.311598 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:27:44.311663 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.311664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.311676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.311747 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.312116 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.312709 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.313411 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.313644 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.313723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.315728 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.315730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.315730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.315739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.317226 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.317598 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.317600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.317600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.317660 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.317859 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.317860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.317861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.317981 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.318055 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.318218 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.318298 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:27:44.318771 15043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:27:44.319152 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.319048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48628)
0925 15:27:44.319098 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.319125 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:27:44.319581 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.319498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48630)
0925 15:27:44.319543 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.319565 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:27:44.319686 15043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:39449: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
I20260925 15:27:44.322500 15118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } attempt: 1
I20260925 15:27:44.322906 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.322793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48646)
0925 15:27:44.322837 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.322873 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:27:44.323374 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.323273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48656)
0925 15:27:44.323325 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.323350 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:27:44.323465 15118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:39449: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
I20260925 15:27:44.350848 15043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } attempt: 1
I20260925 15:27:44.351395 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.351216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48660)
0925 15:27:44.351299 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.351364 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:27:44.351729 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.351635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48664)
0925 15:27:44.351681 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.351707 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:27:44.351823 15043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:39449: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
W20260925 15:27:44.367290 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:44.367543 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:44.367580 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:44.368837 15125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:44.368897 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:44.368922 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:44.368945 15125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:44.368975 15125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:44.370558 15125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:46157,127.1.163.125:39167,127.1.163.124:39449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.163.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:44.370841 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:44.371075 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:44.376297 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:44.376284 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:44.376317 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:44.376318 15125 server_base.cc:1034] running on GCE node
I20260925 15:27:44.376785 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:44.377097 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:44.378274 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790350064378255 us; error 36 us; skew 500 ppm
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:44.380962 15125 init.cc:377] Logged in from keytab as kudu/127.1.163.124@KRBTEST.COM (short username kudu)
I20260925 15:27:44.381460 15125 webserver.cc:460] Webserver started at http://127.1.163.124:40377/ using document root <none> and password file <none>
I20260925 15:27:44.381688 15125 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:44.381758 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:44.381928 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:44.383009 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data/instance:
uuid: "962ca6c1081f40198327c36e69315f62"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "2943f51a4d748cb8ef46828bb82adb4f"
server_key_iv: "3d1ea21ac702eb0b65e95c58c52313de"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.383378 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal/instance:
uuid: "962ca6c1081f40198327c36e69315f62"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "2943f51a4d748cb8ef46828bb82adb4f"
server_key_iv: "3d1ea21ac702eb0b65e95c58c52313de"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.384863 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:27:44.385600 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:44.385808 15125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:27:44.385910 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "962ca6c1081f40198327c36e69315f62"
format_stamp: "Formatted at 2026-09-25 15:27:44 on dist-test-slave-hvv5"
server_key: "2943f51a4d748cb8ef46828bb82adb4f"
server_key_iv: "3d1ea21ac702eb0b65e95c58c52313de"
server_key_version: "encryptionkey@0"
I20260925 15:27:44.385998 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:44.391616 15118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } attempt: 2
I20260925 15:27:44.392050 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.391921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48666)
0925 15:27:44.391982 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.392021 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:27:44.392402 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.392306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48682)
0925 15:27:44.392352 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.392379 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:27:44.392490 15118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:39449: Network error: Client connection negotiation failed: client connection to 127.1.163.124:39449: connect: Connection refused (error 111)
I20260925 15:27:44.399621 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:44.402088 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:44.402261 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:44.406404 15125 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.124:39449
I20260925 15:27:44.406466 15192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.124:39449 every 8 connection(s)
I20260925 15:27:44.406843 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
I20260925 15:27:44.407974 15193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:44.409650 15193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:44.416464 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15125
I20260925 15:27:44.416600 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349904034090-1677-0/minicluster-data/master-2/wal/instance
I20260925 15:27:44.416836 1677 external_mini_cluster.cc:1428] Setting key 0369df30675ea692c56ca8a19200f165
I20260925 15:27:44.419219 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.410445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.1.163.126:46157)
0925 15:27:44.410532 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.410534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.410745 (+ 211us) server_negotiation.cc:407] Connection header received
0925 15:27:44.411193 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.411194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.411204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.411266 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.411902 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.412385 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.413419 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.413637 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.413698 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.415761 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.415762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.415763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.415770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.417489 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.418053 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.418054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.418054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.418113 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.418521 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.418523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.418523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.418670 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.418747 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.419031 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.419120 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:27:44.419310 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.410426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:46157 (local address 127.0.0.1:55862)
0925 15:27:44.410611 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.410649 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.411104 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.411348 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.411352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.411610 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.411805 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.411810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.412467 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.412468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.413336 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.413339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.413715 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.413716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.413819 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.414310 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.414318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.415659 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.417577 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.417583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.417584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.417944 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.418190 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.418191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.418191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.418350 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.418759 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.418765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.418887 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.418988 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.419114 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260925 15:27:44.419989 15193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:27:44.426941 15043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } attempt: 2
I20260925 15:27:44.427183 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.417264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.1.163.124:39449)
0925 15:27:44.417490 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.417496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.417514 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:27:44.417537 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.417539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.417554 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.417626 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.418518 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.419541 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.420645 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.420960 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.421052 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.423847 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.423857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.423857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.423865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.425887 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.426230 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.426233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.426233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.426295 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.426492 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.426493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.426494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.426672 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.426787 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.426967 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.427056 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
I20260925 15:27:44.428057 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.420390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.1.163.125:39167)
0925 15:27:44.420444 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.420447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.420456 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:27:44.420483 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.420484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.420495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.420573 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.420869 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.421377 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.422500 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.422742 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.422825 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.424971 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.424972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.424972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.424978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.426539 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.426919 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.426920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.426920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.426981 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.427479 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.427480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.427480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.427625 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.427669 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.427893 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.427969 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:27:44.428495 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.420312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:39167 (local address 127.0.0.1:50906)
0925 15:27:44.420365 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.420376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.420415 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.420576 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.420577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.420731 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.420778 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.420780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.421455 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.421456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.422403 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.422406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.422851 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.422852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.422912 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.423332 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.423335 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.424882 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.426620 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.426620 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.426621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.426823 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.427107 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.427108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.427108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.427164 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.427718 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.427719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.427759 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.428320 (+ 561us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.428409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:27:44.429070 15193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:27:44.436040 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.427227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48700)
0925 15:27:44.427386 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.427395 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.427432 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.427570 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.427572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.427736 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.427785 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.427788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.428763 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.428764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.429529 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.429531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.429970 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.429971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.430033 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.430469 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.430474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.431936 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.434385 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.434386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.434386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.435102 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.435307 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.435308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.435309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.435377 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.435672 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.435674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.435726 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.435830 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.435936 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 15:27:44.436040 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.427257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.1.163.124:39449)
0925 15:27:44.427297 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.427298 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.427440 (+ 142us) server_negotiation.cc:407] Connection header received
0925 15:27:44.427465 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.427466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.427477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.427852 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.427880 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.428676 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.429604 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.429898 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.429971 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.432023 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.432024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.432024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.432031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.434283 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.435174 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.435175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.435175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.435227 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.435445 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.435446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.435446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.435578 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.435672 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.435875 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.435948 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:27:44.437009 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.428120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.1.163.126:46157)
0925 15:27:44.428168 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.428171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.428207 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:27:44.428282 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.428282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.428292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.428339 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.428984 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.429682 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.430652 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.430919 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.431000 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.433506 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.433508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.433508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.433515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.434995 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.435373 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.435374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.435374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.435425 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.435622 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.435623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.435623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.435751 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.436012 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.436861 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.436936 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:27:44.437523 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.429341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48710)
0925 15:27:44.429618 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.429628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.429669 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.429820 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.429821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.429962 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.429993 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.429994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.430845 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.430846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.431550 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.431553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.432135 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.432136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.432193 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.433101 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.433106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.434593 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.436452 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.436452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.436453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.436675 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.436897 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.436898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.436899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.436952 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.437224 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.437225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.437273 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.437347 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.437453 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260925 15:27:44.437687 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.429369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.1.163.124:39449)
0925 15:27:44.429531 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.429532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.429676 (+ 144us) server_negotiation.cc:407] Connection header received
0925 15:27:44.429715 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.429716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.429725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.429797 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.430080 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.430760 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.431637 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.431994 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.432081 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.434699 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.434700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.434700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.434706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.436388 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.436768 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.436769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.436770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.436823 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.437022 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.437023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.437024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.437157 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.437233 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.437392 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.437527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":144,"thread_start_us":118,"threads_started":1}
I20260925 15:27:44.438186 15043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: Bootstrap starting.
I20260925 15:27:44.438822 15043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:44.438987 15193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: Bootstrap starting.
I20260925 15:27:44.439154 15043 log.cc:824] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:44.439678 15193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:44.439975 15193 log.cc:824] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:44.440050 15043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: No bootstrap required, opened a new log
I20260925 15:27:44.441932 15043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.442070 15043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:44.442113 15043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300d77fb441f41aa9514433b9b77ef45, State: Initialized, Role: FOLLOWER
I20260925 15:27:44.442231 15043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.442508 15201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.442569 15043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:44.442588 15201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
W20260925 15:27:44.444481 15212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 15:27:44.444526 15193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: No bootstrap required, opened a new log
W20260925 15:27:44.444556 15212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:27:44.446985 15193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.447144 15193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:44.447190 15193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962ca6c1081f40198327c36e69315f62, State: Initialized, Role: FOLLOWER
I20260925 15:27:44.447335 15193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.447675 15193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:44.445279 14975 master_runner.cc:186] Error getting master registration for 127.1.163.125:39167: OK, instance_id { permanent_uuid: "24533a49bffe439d9264b9377930b871" instance_seqno: 1790350064247596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:27:44.448149 14975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:27:44.448329 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.448421 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.449029 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.440005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.1.163.125:39167)
0925 15:27:44.440053 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.440055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.440064 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:27:44.440090 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.440091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.440103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.440162 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.440742 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.441461 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.442141 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.442409 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.442486 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.444981 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.444982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.444983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.444992 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.446788 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.447192 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.447193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.447194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.447250 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.447448 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.447449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.447449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.447590 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.448232 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.448787 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.448888 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 15:27:44.449723 15225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:27:44.449777 15225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:27:44.450335 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:27:44.451229 15125 master_runner.cc:186] Error getting master registration for 127.1.163.125:39167: OK, instance_id { permanent_uuid: "24533a49bffe439d9264b9377930b871" instance_seqno: 1790350064247596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:27:44.454138 15125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:27:44.466893 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.457273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.1.163.124:39449)
0925 15:27:44.457317 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.457319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.457796 (+ 477us) server_negotiation.cc:407] Connection header received
0925 15:27:44.457897 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.457898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.457911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.457941 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.458909 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.459727 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.460875 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.461288 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.461349 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.463534 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.463535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.463535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.463543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.465251 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.465998 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.466000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.466000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.466057 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.466249 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.466250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.466250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.466386 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.466490 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.466706 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.466793 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:27:44.468705 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.456936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.1.163.125:39167)
0925 15:27:44.456983 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.456985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.457481 (+ 496us) server_negotiation.cc:407] Connection header received
0925 15:27:44.457591 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.457592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.457603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.457633 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.458867 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.459660 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.460793 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.461136 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.461182 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.464206 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.464207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.464207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.464214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.466609 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.467772 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.467773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.467773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.467842 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.468054 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.468055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.468055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.468185 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.468270 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.468486 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.468611 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:27:44.470887 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.457503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.1.163.126:46157)
0925 15:27:44.457554 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.457556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.458717 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 15:27:44.461879 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.461881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.461893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.461958 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.462723 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.463451 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.465226 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.465538 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.465579 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.467926 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.467927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.467927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.467934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.469678 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.470079 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.470080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.470080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.470140 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.470350 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.470351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.470351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.470479 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.470569 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.470716 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.470794 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:27:44.483628 15118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } attempt: 3
Sep 25 15:27:44 dist-test-slave-hvv5 krb5kdc[14962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:27:44.492777 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.484005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.1.163.124:39449)
0925 15:27:44.484048 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:44.484050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:44.484059 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:27:44.484114 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:44.484115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:44.484125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:44.484184 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:44.484576 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.485355 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.486353 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.486677 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.486752 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.489262 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:44.489264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:44.489264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:44.489272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:44.491475 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.491887 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.491888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.491889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.491954 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:44.492168 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:44.492169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:44.492169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:44.492340 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:44.492411 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:44.492599 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:27:44.492683 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:27:44.492776 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:44.483922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:39449 (local address 127.0.0.1:48732)
0925 15:27:44.483993 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:44.484005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:44.484055 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:44.484195 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:44.484196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:44.484409 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:44.484465 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.484467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.485449 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.485451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.486259 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:44.486261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:44.486779 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:44.486780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:44.486849 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:44.487432 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:44.487437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:44.489127 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:44.491570 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.491571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.491571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.491807 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.492029 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:44.492030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:44.492030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:44.492099 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:44.492417 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:44.492417 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:44.492466 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:27:44.492559 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:27:44.492665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:27:44.495064 15118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: Bootstrap starting.
I20260925 15:27:44.495858 15118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:44.496163 15118 log.cc:824] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:44.497130 15118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: No bootstrap required, opened a new log
I20260925 15:27:44.499166 15118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.499327 15118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:44.499364 15118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24533a49bffe439d9264b9377930b871, State: Initialized, Role: FOLLOWER
I20260925 15:27:44.499508 15118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.499792 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.499841 15118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:27:44.501780 15246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:27:44.501838 15246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:27:44.501415 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.503764 15050 master_runner.cc:263] No leader master found from master 24533a49bffe439d9264b9377930b871
I20260925 15:27:44.503818 15050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:27:44.554852 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:44.555018 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.555296 15235 leader_election.cc:288] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 300d77fb441f41aa9514433b9b77ef45 (127.1.163.126:46157), 962ca6c1081f40198327c36e69315f62 (127.1.163.124:39449)
I20260925 15:27:44.555564 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24533a49bffe439d9264b9377930b871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300d77fb441f41aa9514433b9b77ef45" is_pre_election: true
I20260925 15:27:44.555739 15017 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24533a49bffe439d9264b9377930b871 in term 0.
I20260925 15:27:44.555989 15067 leader_election.cc:302] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24533a49bffe439d9264b9377930b871, 300d77fb441f41aa9514433b9b77ef45; no voters:
I20260925 15:27:44.556267 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:27:44.556348 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:27:44.556391 15235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:44.557435 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.557570 15235 leader_election.cc:288] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [CANDIDATE]: Term 1 election: Requested vote from peers 300d77fb441f41aa9514433b9b77ef45 (127.1.163.126:46157), 962ca6c1081f40198327c36e69315f62 (127.1.163.124:39449)
I20260925 15:27:44.557793 15017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24533a49bffe439d9264b9377930b871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300d77fb441f41aa9514433b9b77ef45"
I20260925 15:27:44.557881 15017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:44.558854 15017 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24533a49bffe439d9264b9377930b871 in term 1.
I20260925 15:27:44.559041 15067 leader_election.cc:302] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24533a49bffe439d9264b9377930b871, 300d77fb441f41aa9514433b9b77ef45; no voters:
I20260925 15:27:44.560078 15167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24533a49bffe439d9264b9377930b871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962ca6c1081f40198327c36e69315f62" is_pre_election: true
I20260925 15:27:44.560261 15167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24533a49bffe439d9264b9377930b871 in term 0.
I20260925 15:27:44.560079 15166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24533a49bffe439d9264b9377930b871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962ca6c1081f40198327c36e69315f62"
I20260925 15:27:44.560437 15166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:44.561385 15166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24533a49bffe439d9264b9377930b871 in term 1.
I20260925 15:27:44.561944 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:44.562129 15235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [term 1 LEADER]: Becoming Leader. State: Replica: 24533a49bffe439d9264b9377930b871, State: Running, Role: LEADER
I20260925 15:27:44.562234 15235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } }
I20260925 15:27:44.562665 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24533a49bffe439d9264b9377930b871. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.562748 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:44.562978 15250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:44.563187 15250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:44.565442 15166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [term 1 FOLLOWER]: Refusing update from remote peer 24533a49bffe439d9264b9377930b871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:44.565769 15235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:44.567101 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24533a49bffe439d9264b9377930b871. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.567196 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.567559 15017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [term 1 FOLLOWER]: Refusing update from remote peer 24533a49bffe439d9264b9377930b871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:44.567834 15235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:44.569010 15201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24533a49bffe439d9264b9377930b871. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.569124 15201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.570520 15251 mvcc.cc:205] Tried to move back new op lower bound from 7333273864457650176 to 7333273864447705088. Current Snapshot: MvccSnapshot[applied={T|T < 7333273864457650176}]
I20260925 15:27:44.570804 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.570888 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:44.571352 15201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.571457 15201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.571791 15254 mvcc.cc:205] Tried to move back new op lower bound from 7333273864457650176 to 7333273864447705088. Current Snapshot: MvccSnapshot[applied={T|T < 7333273864457650176}]
I20260925 15:27:44.572702 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.572790 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:44.573204 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24533a49bffe439d9264b9377930b871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "300d77fb441f41aa9514433b9b77ef45" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 46157 } } peers { permanent_uuid: "24533a49bffe439d9264b9377930b871" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 39167 } } peers { permanent_uuid: "962ca6c1081f40198327c36e69315f62" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 39449 } } }
I20260925 15:27:44.573328 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:27:44.574025 15252 mvcc.cc:205] Tried to move back new op lower bound from 7333273864457650176 to 7333273864447705088. Current Snapshot: MvccSnapshot[applied={T|T < 7333273864457650176}]
I20260925 15:27:44.576192 15250 catalog_manager.cc:1347] Generated new cluster ID: 92ed594c3bf0479694dee941f7abed02
I20260925 15:27:44.576252 15250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:27:44.596849 15250 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:27:44.597618 15250 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:44.610180 15250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24533a49bffe439d9264b9377930b871: Generated new TSK 0
I20260925 15:27:44.610410 15250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:27:45.445019 15212 catalog_manager.cc:1259] Loaded cluster ID: 92ed594c3bf0479694dee941f7abed02
I20260925 15:27:45.445096 15212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: loading cluster ID for follower catalog manager: success
I20260925 15:27:45.445945 15212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: acquiring CA information for follower catalog manager: success
I20260925 15:27:45.446262 15212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 300d77fb441f41aa9514433b9b77ef45: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:27:45.450206 15225 catalog_manager.cc:1259] Loaded cluster ID: 92ed594c3bf0479694dee941f7abed02
I20260925 15:27:45.450269 15225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: loading cluster ID for follower catalog manager: success
I20260925 15:27:45.451150 15225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: acquiring CA information for follower catalog manager: success
I20260925 15:27:45.451442 15225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 962ca6c1081f40198327c36e69315f62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:27:45.670961 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 14975
I20260925 15:27:45.676457 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15050
I20260925 15:27:45.681545 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15125
2026-09-25T15:27:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2944 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:27:45 dist-test-slave-hvv5 krb5kdc[15265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:27:45 dist-test-slave-hvv5 krb5kdc[15265](info): set up 2 sockets
Sep 25 15:27:45 dist-test-slave-hvv5 krb5kdc[15265](info): commencing operation
krb5kdc: starting...
W20260925 15:27:46.740897 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:27:46 dist-test-slave-hvv5 krb5kdc[15265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:27:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:27:46.875507 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37875
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:46.960829 15278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:46.961056 15278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:46.961079 15278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:46.962321 15278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:46.962373 15278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:46.962387 15278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:46.962400 15278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:46.962412 15278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:46.963994 15278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:46.964255 15278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:46.964529 15278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:46.969656 15284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:46.969656 15286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:46.969735 15283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:46.969833 15278 server_base.cc:1034] running on GCE node
I20260925 15:27:46.970196 15278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:46.970497 15278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:46.971674 15278 hybrid_clock.cc:648] HybridClock initialized: now 1790350066971674 us; error 46 us; skew 500 ppm
Sep 25 15:27:46 dist-test-slave-hvv5 krb5kdc[15265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350066, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:46.974344 15278 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:46.974830 15278 webserver.cc:460] Webserver started at http://127.1.163.126:35859/ using document root <none> and password file <none>
I20260925 15:27:46.975081 15278 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:46.975152 15278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:46.975297 15278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:46.976408 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "467040524c0343ee8d02b7a09d1df5a7"
format_stamp: "Formatted at 2026-09-25 15:27:46 on dist-test-slave-hvv5"
server_key: "2b6050b6107135f5d2a1d023e916a061"
server_key_iv: "24266f2b34cdbb7931aea4e86e896109"
server_key_version: "encryptionkey@0"
I20260925 15:27:46.976835 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "467040524c0343ee8d02b7a09d1df5a7"
format_stamp: "Formatted at 2026-09-25 15:27:46 on dist-test-slave-hvv5"
server_key: "2b6050b6107135f5d2a1d023e916a061"
server_key_iv: "24266f2b34cdbb7931aea4e86e896109"
server_key_version: "encryptionkey@0"
I20260925 15:27:46.978278 15278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:46.979240 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:46.979532 15278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:46.979650 15278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "467040524c0343ee8d02b7a09d1df5a7"
format_stamp: "Formatted at 2026-09-25 15:27:46 on dist-test-slave-hvv5"
server_key: "2b6050b6107135f5d2a1d023e916a061"
server_key_iv: "24266f2b34cdbb7931aea4e86e896109"
server_key_version: "encryptionkey@0"
I20260925 15:27:46.979734 15278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:47.002643 15278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:47.005564 15278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:47.005780 15278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:47.010318 15278 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37875
I20260925 15:27:47.010404 15345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37875 every 8 connection(s)
I20260925 15:27:47.010828 15278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:47.011130 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15278
I20260925 15:27:47.011255 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:27:47.011490 1677 external_mini_cluster.cc:1428] Setting key 014a7a9c3a5b1fdff88bfa09c33c8a4b
I20260925 15:27:47.011955 15346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:47.015735 15346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7: Bootstrap starting.
I20260925 15:27:47.016515 15346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:47.016912 15346 log.cc:824] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:47.017819 15346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7: No bootstrap required, opened a new log
Sep 25 15:27:47 dist-test-slave-hvv5 krb5kdc[15265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:47.020469 15346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } }
I20260925 15:27:47.020709 15346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:47.020751 15346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467040524c0343ee8d02b7a09d1df5a7, State: Initialized, Role: FOLLOWER
I20260925 15:27:47.020895 15346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } }
I20260925 15:27:47.021023 15346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:47.021059 15346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:47.021108 15346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:47.021972 15346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } }
I20260925 15:27:47.022089 15346 leader_election.cc:302] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467040524c0343ee8d02b7a09d1df5a7; no voters:
I20260925 15:27:47.022260 15346 leader_election.cc:288] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:27:47.022356 15351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:47.022537 15346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:47.022557 15351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [term 1 LEADER]: Becoming Leader. State: Replica: 467040524c0343ee8d02b7a09d1df5a7, State: Running, Role: LEADER
I20260925 15:27:47.022735 15351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } }
I20260925 15:27:47.023134 15353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467040524c0343ee8d02b7a09d1df5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } } }
I20260925 15:27:47.023327 15353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:47.023132 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467040524c0343ee8d02b7a09d1df5a7. Latest consensus state: current_term: 1 leader_uuid: "467040524c0343ee8d02b7a09d1df5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467040524c0343ee8d02b7a09d1df5a7" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37875 } } }
I20260925 15:27:47.023556 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:47.023543 15360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:47.024482 15360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:47.024762 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:47.012127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.1.163.126:37875)
0925 15:27:47.012348 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:47.012358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:47.012395 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:27:47.013201 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:47.013209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:47.013262 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:47.013385 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:47.014377 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:47.015322 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:47.016408 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:47.016959 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:47.017107 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:47.020027 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:47.020033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:47.020034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:47.020051 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:47.022323 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:47.022825 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:47.022827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:47.022828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:47.022980 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:47.023255 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:47.023256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:47.023256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:47.023602 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:47.023764 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:47.024361 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 15:27:47.024506 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260925 15:27:47.026113 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:27:47.026362 15360 catalog_manager.cc:1347] Generated new cluster ID: 9b12bf948f364c43ab5c5f4f6c0d6306
I20260925 15:27:47.026410 15360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:27:47.059559 15360 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:27:47.060228 15360 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:47.063179 15360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467040524c0343ee8d02b7a09d1df5a7: Generated new TSK 0
I20260925 15:27:47.063378 15360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:27:47.080315 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15278
2026-09-25T15:27:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1394 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:27:47 dist-test-slave-hvv5 krb5kdc[15373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:27:47 dist-test-slave-hvv5 krb5kdc[15373](info): set up 2 sockets
Sep 25 15:27:47 dist-test-slave-hvv5 krb5kdc[15373](info): commencing operation
krb5kdc: starting...
W20260925 15:27:48.124681 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:27:48.264061 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:48.348665 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:48.348887 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:48.348912 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:48.350059 15386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:48.350095 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:48.350108 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:48.350121 15386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:48.350133 15386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:48.351563 15386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:48.351774 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:48.351949 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:48.357388 15394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.357376 15392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.357436 15386 server_base.cc:1034] running on GCE node
W20260925 15:27:48.357376 15391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.357919 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:48.358227 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:48.359385 15386 hybrid_clock.cc:648] HybridClock initialized: now 1790350068359373 us; error 33 us; skew 500 ppm
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:48.361979 15386 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:48.362447 15386 webserver.cc:460] Webserver started at http://127.1.163.126:33023/ using document root <none> and password file <none>
I20260925 15:27:48.362634 15386 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:48.362675 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:48.362774 15386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:48.363801 15386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "e25110f5730145da8353888a96572c71"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "7699fb66fc5e614d8e6a0c8f5a7ff34b"
server_key_iv: "938b7c4e3909785f63661559b56d267c"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.364108 15386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "e25110f5730145da8353888a96572c71"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "7699fb66fc5e614d8e6a0c8f5a7ff34b"
server_key_iv: "938b7c4e3909785f63661559b56d267c"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.365586 15386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:27:48.366497 15401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.366741 15386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:48.366864 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "e25110f5730145da8353888a96572c71"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "7699fb66fc5e614d8e6a0c8f5a7ff34b"
server_key_iv: "938b7c4e3909785f63661559b56d267c"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.366958 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:48.386394 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:48.388659 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:48.388881 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:48.393069 15386 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44899
I20260925 15:27:48.393137 15453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44899 every 8 connection(s)
I20260925 15:27:48.393509 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:48.394541 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:48.397018 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap starting.
I20260925 15:27:48.397658 15454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:48.397974 15454 log.cc:824] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:48.398734 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: No bootstrap required, opened a new log
I20260925 15:27:48.399565 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15386
I20260925 15:27:48.399673 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:27:48.399875 1677 external_mini_cluster.cc:1428] Setting key 5cb3d14cd6744b67a44026a57055d961
I20260925 15:27:48.401276 15454 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:48.401458 15454 raft_consensus.cc:374] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:48.401495 15454 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25110f5730145da8353888a96572c71, State: Initialized, Role: FOLLOWER
I20260925 15:27:48.401631 15454 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:48.401718 15454 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:48.401751 15454 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:48.401819 15454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:48.402858 15454 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:48.402949 15454 leader_election.cc:302] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25110f5730145da8353888a96572c71; no voters:
I20260925 15:27:48.403131 15454 leader_election.cc:288] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:27:48.403172 15459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:48.403298 15459 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 LEADER]: Becoming Leader. State: Replica: e25110f5730145da8353888a96572c71, State: Running, Role: LEADER
I20260925 15:27:48.403380 15459 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:48.403411 15454 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:48.403750 15459 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25110f5730145da8353888a96572c71. Latest consensus state: current_term: 1 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:48.403821 15459 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:48.403771 15460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:48.403999 15460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:48.404781 15463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:48.405382 15463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:48.406876 15463 catalog_manager.cc:1347] Generated new cluster ID: 78a7e0c17be94d8b862e01287536a727
I20260925 15:27:48.406931 15463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:48.414199 15463 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:27:48.414706 15463 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:48.415705 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.400693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.1.163.126:44899)
0925 15:27:48.400884 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.400893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.400919 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:27:48.401259 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.401265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.401323 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.401439 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:48.402233 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.403306 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.404404 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.405062 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.405214 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.411252 (+ 6038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:48.411257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:48.411257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:48.411272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:48.413470 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.413945 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.413949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.413949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.414067 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.414343 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.414344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.414345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.414642 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:48.414814 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.415310 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.415468 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260925 15:27:48.421003 15463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Generated new TSK 0
I20260925 15:27:48.421209 15463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:27:48.457405 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:48.546433 15480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:48.546694 15480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:48.546721 15480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:48.547981 15480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:48.548031 15480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:48.548061 15480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:27:48.549695 15480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:48.549993 15480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:48.550246 15480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:48.555694 15485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.555717 15480 server_base.cc:1034] running on GCE node
W20260925 15:27:48.555694 15488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.555711 15486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.556167 15480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:48.556442 15480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:48.557600 15480 hybrid_clock.cc:648] HybridClock initialized: now 1790350068557599 us; error 42 us; skew 500 ppm
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:48.560314 15480 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:27:48.560858 15480 webserver.cc:460] Webserver started at http://127.1.163.65:33595/ using document root <none> and password file <none>
I20260925 15:27:48.561065 15480 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:48.561107 15480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:48.561203 15480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:48.562243 15480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "dcb33d5bc0c5abf3564b479084180662"
server_key_iv: "c5d7a8a6f1a70f53b73b4e19efd01d82"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.562549 15480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "dcb33d5bc0c5abf3564b479084180662"
server_key_iv: "c5d7a8a6f1a70f53b73b4e19efd01d82"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.563915 15480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:27:48.564889 15495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.565109 15480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:48.565208 15480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "dcb33d5bc0c5abf3564b479084180662"
server_key_iv: "c5d7a8a6f1a70f53b73b4e19efd01d82"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.565263 15480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:48.579689 15480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:48.582239 15480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:48.582373 15480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:48.582562 15480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:48.582919 15480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:48.582957 15480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.582980 15480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:48.582993 15480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.589027 15480 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35539
I20260925 15:27:48.589110 15608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35539 every 8 connection(s)
I20260925 15:27:48.589449 15480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:27:48.593221 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15480
I20260925 15:27:48.593349 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:27:48.593557 1677 external_mini_cluster.cc:1428] Setting key f6991771eaef81d97c616dbaae322c48
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:48.600456 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.590265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:45619 (local address 127.1.163.126:44899)
0925 15:27:48.590412 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.590414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.590567 (+ 153us) server_negotiation.cc:407] Connection header received
0925 15:27:48.591076 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.591077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.591088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.591150 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:48.592161 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.592783 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.594220 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.594467 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.594523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.597095 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:48.597096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:48.597097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:48.597105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:48.598690 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.599248 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.599250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.599250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.599315 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.599666 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.599667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.599667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.599835 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:48.599891 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.600294 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.600385 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 15:27:48.600802 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.590248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:45619)
0925 15:27:48.590426 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.590471 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.590962 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.591225 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.591228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.591839 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:48.592039 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.592044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.592871 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.592871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.594104 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.594107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.594554 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.594555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.594711 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.595313 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:48.595331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:48.596987 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:48.598784 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.598793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.598796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.599130 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.599400 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.599401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.599402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.599592 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.599908 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:48.599917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:48.600119 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.600259 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.600472 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260925 15:27:48.601377 15609 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:48.601506 15609 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:48.601720 15609 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:48.602299 15418 ts_manager.cc:194] Registered new tserver with Master: 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539)
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:27:48.602986 15418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:45619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:27:48.624293 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:48.708845 15614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:48.709103 15614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:48.709143 15614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:48.710376 15614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:48.710456 15614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:48.710505 15614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:27:48.712080 15614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:48.712384 15614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:48.712664 15614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:48.718179 15622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.718277 15620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.718179 15619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.718485 15614 server_base.cc:1034] running on GCE node
I20260925 15:27:48.718719 15614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:48.719025 15614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:48.720254 15614 hybrid_clock.cc:648] HybridClock initialized: now 1790350068720228 us; error 46 us; skew 500 ppm
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:48.723054 15614 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:27:48.723515 15614 webserver.cc:460] Webserver started at http://127.1.163.66:33283/ using document root <none> and password file <none>
I20260925 15:27:48.723759 15614 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:48.723838 15614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:48.723990 15614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:48.725144 15614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "84cedc26793d4a119bd1c0317720933a"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "d5f6122617a32a3b0008bfc8031933c1"
server_key_iv: "2aa63a26c44a9d87504a9de94e4a09f2"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.725502 15614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "84cedc26793d4a119bd1c0317720933a"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "d5f6122617a32a3b0008bfc8031933c1"
server_key_iv: "2aa63a26c44a9d87504a9de94e4a09f2"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.726951 15614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:27:48.727856 15629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.728097 15614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:48.728214 15614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "84cedc26793d4a119bd1c0317720933a"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "d5f6122617a32a3b0008bfc8031933c1"
server_key_iv: "2aa63a26c44a9d87504a9de94e4a09f2"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.728291 15614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:48.737962 15614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:48.740446 15614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:48.740646 15614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:48.740896 15614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:48.741232 15614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:48.741286 15614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.741328 15614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:48.741356 15614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.747479 15614 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34579
I20260925 15:27:48.747522 15742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34579 every 8 connection(s)
I20260925 15:27:48.747855 15614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:27:48.749945 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15614
I20260925 15:27:48.750092 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:27:48.750288 1677 external_mini_cluster.cc:1428] Setting key ffdc380c3d8900112a2295e2293319eb
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:48.759434 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.748638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:45577 (local address 127.1.163.126:44899)
0925 15:27:48.748689 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.748692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.748964 (+ 272us) server_negotiation.cc:407] Connection header received
0925 15:27:48.749464 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.749465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.749476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.749537 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:48.750639 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.751327 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.752631 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.752841 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.752932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.755887 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:48.755890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:48.755890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:48.755900 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:48.757827 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.758357 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.758359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.758359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.758431 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.758727 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.758729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.758729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.758859 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:48.758941 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.759239 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.759331 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
I20260925 15:27:48.759668 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.748663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:45577)
0925 15:27:48.748838 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.748873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.749346 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.749606 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.749610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.750264 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:48.750528 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.750535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.751485 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.751486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.752486 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.752489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.752945 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.752946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.753097 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.753766 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:48.753788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:48.755760 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:48.757964 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.757977 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.757980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.758264 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.758508 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.758509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.758510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.758639 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.758931 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:48.758938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:48.759101 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.759226 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.759410 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 15:27:48.760151 15743 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:48.760258 15743 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:48.760483 15743 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:48.761034 15418 ts_manager.cc:194] Registered new tserver with Master: 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
I20260925 15:27:48.761469 15418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:45577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:27:48.781307 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:48.863924 15748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:48.864153 15748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:48.864178 15748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:48.865432 15748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:48.865484 15748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:48.865512 15748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:27:48.867034 15748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:48.867304 15748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:48.867576 15748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:48.873224 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.873235 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:48.873227 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:48.873512 15748 server_base.cc:1034] running on GCE node
I20260925 15:27:48.873736 15748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:48.874053 15748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:48.875229 15748 hybrid_clock.cc:648] HybridClock initialized: now 1790350068875210 us; error 31 us; skew 500 ppm
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:48.877866 15748 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:27:48.878269 15748 webserver.cc:460] Webserver started at http://127.1.163.67:43069/ using document root <none> and password file <none>
I20260925 15:27:48.878460 15748 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:48.878500 15748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:48.878589 15748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:27:48.879631 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "ab9529b351d9764573a0ffdadc1e7d52"
server_key_iv: "4d20031d53cb546443ade617baf899d0"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.879967 15748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "ab9529b351d9764573a0ffdadc1e7d52"
server_key_iv: "4d20031d53cb546443ade617baf899d0"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.881328 15748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 15:27:48.882002 15763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.882172 15748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:27:48.882256 15748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "ab9529b351d9764573a0ffdadc1e7d52"
server_key_iv: "4d20031d53cb546443ade617baf899d0"
server_key_version: "encryptionkey@0"
I20260925 15:27:48.882313 15748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:48.928776 15748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:48.931319 15748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:48.931445 15748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:48.931648 15748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:48.931980 15748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:48.932022 15748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.932046 15748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:48.932060 15748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:48.938212 15748 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:43173
I20260925 15:27:48.938252 15876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:43173 every 8 connection(s)
I20260925 15:27:48.938593 15748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:27:48 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:48.947347 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15748
I20260925 15:27:48.947459 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:27:48.947633 1677 external_mini_cluster.cc:1428] Setting key 81bf03997bf35c6f598ad5f0f6345778
I20260925 15:27:48.948594 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.939450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:33871 (local address 127.1.163.126:44899)
0925 15:27:48.939539 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.939541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.939687 (+ 146us) server_negotiation.cc:407] Connection header received
0925 15:27:48.940223 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.940225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.940235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.940312 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:48.941305 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.941858 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.942883 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.943084 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.943150 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.945428 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:48.945430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:48.945430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:48.945437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:48.946919 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.947345 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.947347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.947348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.947425 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.947760 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.947762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.947762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.947933 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:48.948024 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.948398 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.948486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:27:48.948992 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.939388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:33871)
0925 15:27:48.939557 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.939594 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.940127 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.940345 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.940348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.940984 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:48.941186 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.941191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.941960 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.941961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.942776 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.942778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.943171 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.943172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.943293 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.943800 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:48.943809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:48.945334 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:48.947010 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.947017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.947019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.947251 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.947531 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:48.947532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:48.947533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:48.947685 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:48.948036 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:48.948043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:48.948211 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.948350 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.948584 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260925 15:27:48.949523 15877 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:48.949620 15877 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:48.949764 15877 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:48.950086 15418 ts_manager.cc:194] Registered new tserver with Master: b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173)
I20260925 15:27:48.950435 15418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:33871
I20260925 15:27:48.959941 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:27:48.968411 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.961620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.1.163.126:44899)
0925 15:27:48.961689 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.961691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.961820 (+ 129us) server_negotiation.cc:407] Connection header received
0925 15:27:48.961865 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.961866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.961913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.961970 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:48.962594 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.963121 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.963917 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.964116 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.964177 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.965502 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:48.965504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:48.965504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:48.965511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:48.967099 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.967457 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.967458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.967458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.967517 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:48.967760 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:48.967761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:48.967761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:48.967923 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:48.968005 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.968195 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.968267 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:27:48.970315 15418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:27:48.972100 15418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:27:48.978775 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.0.0.1:51208)
0925 15:27:48.974668 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.974708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.974776 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.975147 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.975150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.975152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:48.975287 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.975298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976767 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.976768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.977555 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.977557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978449 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.978450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.978529 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.978543 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.978628 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.978679 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260925 15:27:48.979027 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34579 (local address 127.0.0.1:37490)
0925 15:27:48.974868 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.974879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.974949 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.975155 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.975156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.975158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:48.975287 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.975290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976419 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.976420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.977775 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.977778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978812 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.978813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.978876 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.978881 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.978914 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.978956 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":47,"thread_start_us":89,"threads_started":1}
I20260925 15:27:48.979149 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.0.0.1:33860)
0925 15:27:48.975039 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:48.975047 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:48.975085 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:48.975333 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:48.975334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:48.975335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:48.975377 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.975389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976731 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.976732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.977846 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:48.977848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978843 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.978844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:48.978915 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.978921 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:27:48.978969 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:27:48.979010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":26,"thread_start_us":78,"threads_started":1}
I20260925 15:27:48.979194 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.1.163.67:43173)
0925 15:27:48.975124 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.975129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.975137 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:27:48.975174 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.975176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.975191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.975262 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:48.975476 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976622 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.977929 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978681 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.978743 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.978807 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.979062 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.979106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260925 15:27:48.978804 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.1.163.65:35539)
0925 15:27:48.974959 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.974962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.974972 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:27:48.975014 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.975016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.975035 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.975373 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:48.975467 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976652 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.977648 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978362 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.978438 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.978496 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.978671 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.978723 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":89,"threads_started":1}
I20260925 15:27:48.979425 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:48.974690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.1.163.66:34579)
0925 15:27:48.974876 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:48.974882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:48.974949 (+ 67us) server_negotiation.cc:407] Connection header received
0925 15:27:48.975025 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:48.975028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:48.975043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:48.975547 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:48.975622 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.976277 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.977943 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:48.978719 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:48.979232 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:48.979296 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:48.979337 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:27:48.979368 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260925 15:27:48.979966 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 50fef8c63de640509ee752da7f949a51 (DEFAULT_TABLE table=test-table [id=426ef36959e4474db7ee66da3855782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:48.980067 15677 tablet_service.cc:1467] Processing CreateTablet for tablet 50fef8c63de640509ee752da7f949a51 (DEFAULT_TABLE table=test-table [id=426ef36959e4474db7ee66da3855782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:48.980260 15543 tablet_service.cc:1467] Processing CreateTablet for tablet 50fef8c63de640509ee752da7f949a51 (DEFAULT_TABLE table=test-table [id=426ef36959e4474db7ee66da3855782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:48.980355 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fef8c63de640509ee752da7f949a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:48.980382 15677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fef8c63de640509ee752da7f949a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:48.980568 15543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fef8c63de640509ee752da7f949a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:48.982980 15894 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Bootstrap starting.
I20260925 15:27:48.983718 15895 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap starting.
I20260925 15:27:48.983884 15894 tablet_bootstrap.cc:654] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:48.983937 15896 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Bootstrap starting.
I20260925 15:27:48.984397 15895 tablet_bootstrap.cc:654] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:48.984467 15896 tablet_bootstrap.cc:654] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:48.984807 15894 log.cc:824] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:48.985003 15895 log.cc:824] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:48.985157 15896 log.cc:824] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:48.985661 15895 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: No bootstrap required, opened a new log
I20260925 15:27:48.985671 15894 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: No bootstrap required, opened a new log
I20260925 15:27:48.985737 15894 ts_tablet_manager.cc:1397] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:27:48.985743 15895 ts_tablet_manager.cc:1397] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:27:48.987504 15894 raft_consensus.cc:348] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.987674 15894 raft_consensus.cc:374] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:48.987578 15895 raft_consensus.cc:348] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.987721 15894 raft_consensus.cc:729] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f8fce3e2574c09bcb8a46c70c23f70, State: Initialized, Role: FOLLOWER
I20260925 15:27:48.987748 15895 raft_consensus.cc:374] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:48.987790 15895 raft_consensus.cc:729] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58544f8abe8e4e6b9bfe4389541bccb9, State: Initialized, Role: FOLLOWER
I20260925 15:27:48.987859 15894 consensus_queue.cc:260] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.987905 15895 consensus_queue.cc:260] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.988161 15894 ts_tablet_manager.cc:1428] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:27:48.988184 15895 ts_tablet_manager.cc:1428] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:27:48.988184 15609 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
I20260925 15:27:48.988308 15877 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
I20260925 15:27:48.988919 15896 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: No bootstrap required, opened a new log
I20260925 15:27:48.988997 15896 ts_tablet_manager.cc:1397] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 15:27:48.990509 15896 raft_consensus.cc:348] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.990646 15896 raft_consensus.cc:374] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:48.990669 15896 raft_consensus.cc:729] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cedc26793d4a119bd1c0317720933a, State: Initialized, Role: FOLLOWER
I20260925 15:27:48.990783 15896 consensus_queue.cc:260] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:48.991107 15896 ts_tablet_manager.cc:1428] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 15:27:48.991115 15743 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
W20260925 15:27:48.998487 15744 tablet.cc:2369] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:27:49.090278 15610 tablet.cc:2369] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:27:49.097558 15901 raft_consensus.cc:482] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:49.097697 15901 raft_consensus.cc:504] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.098052 15901 leader_election.cc:288] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579), b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173)
I20260925 15:27:49.101296 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.098510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:52677 (local address 127.1.163.67:43173)
0925 15:27:49.098559 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.098561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.098616 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:27:49.098671 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.098673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.098684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.098753 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.098964 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.099421 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.100453 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.100923 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.101007 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.101018 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.101162 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.101210 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:27:49.101305 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.098284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34579 (local address 127.1.163.65:42223)
0925 15:27:49.098343 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.098358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.098425 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.098626 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.098627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.098629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.098711 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.098713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.099588 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.099589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.100349 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.100352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.101033 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.101034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.101113 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.101116 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.101192 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.101230 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":4}
I20260925 15:27:49.101306 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.098440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.1.163.65:52677)
0925 15:27:49.098560 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.098568 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.098616 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.098814 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.098814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.098816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.098890 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.098892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.099492 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.099493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.100348 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.100350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.101010 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.101011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.101091 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.101094 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.101138 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.101192 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":27,"thread_start_us":70,"threads_started":1}
I20260925 15:27:49.101336 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.098383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42223 (local address 127.1.163.66:34579)
0925 15:27:49.098431 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.098432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.098443 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:27:49.098514 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.098515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.098533 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.098601 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.098810 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.099501 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.100445 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.100927 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.101022 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.101031 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.101214 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.101266 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:27:49.101541 15902 raft_consensus.cc:482] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:49.101649 15902 raft_consensus.cc:504] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.101735 15831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" is_pre_election: true
I20260925 15:27:49.101733 15697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cedc26793d4a119bd1c0317720933a" is_pre_election: true
I20260925 15:27:49.101846 15697 raft_consensus.cc:2454] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58544f8abe8e4e6b9bfe4389541bccb9 in term 0.
I20260925 15:27:49.101874 15831 raft_consensus.cc:2454] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58544f8abe8e4e6b9bfe4389541bccb9 in term 0.
I20260925 15:27:49.102005 15902 leader_election.cc:288] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539), b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173)
I20260925 15:27:49.102126 15498 leader_election.cc:302] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9, b3f8fce3e2574c09bcb8a46c70c23f70; no voters:
I20260925 15:27:49.102236 15901 raft_consensus.cc:2781] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:27:49.102283 15901 raft_consensus.cc:482] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:27:49.102303 15901 raft_consensus.cc:3037] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:49.103410 15901 raft_consensus.cc:504] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.103535 15901 leader_election.cc:288] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 1 election: Requested vote from peers 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579), b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173)
I20260925 15:27:49.104632 15697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84cedc26793d4a119bd1c0317720933a"
I20260925 15:27:49.104657 15831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
I20260925 15:27:49.104728 15831 raft_consensus.cc:3037] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:49.104714 15697 raft_consensus.cc:3037] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:49.105337 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.102123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.1.163.66:36323)
0925 15:27:49.102172 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.102182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.102218 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.102484 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.102485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.102486 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.102547 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.102548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.103426 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.103428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.104221 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.104223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.105042 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.105043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.105128 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.105132 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.105181 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.105226 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:27:49.105499 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.102296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40295 (local address 127.1.163.65:35539)
0925 15:27:49.102336 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.102338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.102444 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:27:49.102544 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.102545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.102555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.102617 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.102805 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.103477 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.104578 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.105115 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.105158 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.105166 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.105383 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.105424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:27:49.105608 15697 raft_consensus.cc:2454] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58544f8abe8e4e6b9bfe4389541bccb9 in term 1.
I20260925 15:27:49.105608 15831 raft_consensus.cc:2454] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58544f8abe8e4e6b9bfe4389541bccb9 in term 1.
I20260925 15:27:49.105608 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.102319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:36323 (local address 127.1.163.67:43173)
0925 15:27:49.102356 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.102358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.102365 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:27:49.102386 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.102387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.102395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.102446 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.102640 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.103345 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.104363 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.104939 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.105005 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.105012 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.105500 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.105544 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:27:49.105719 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.102123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.1.163.66:40295)
0925 15:27:49.102394 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.102404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.102465 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.102659 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.102660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.102662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.102727 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.102728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.103561 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.103563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.104456 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.104458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.105222 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.105222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.105300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.105303 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.105593 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.105636 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":11,"thread_start_us":112,"threads_started":1}
I20260925 15:27:49.105762 15498 leader_election.cc:302] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9, 84cedc26793d4a119bd1c0317720933a; no voters:
I20260925 15:27:49.105836 15831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" is_pre_election: true
I20260925 15:27:49.105870 15901 raft_consensus.cc:2781] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:49.105939 15831 raft_consensus.cc:2379] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84cedc26793d4a119bd1c0317720933a in current term 1: Already voted for candidate 58544f8abe8e4e6b9bfe4389541bccb9 in this term.
I20260925 15:27:49.106042 15563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fef8c63de640509ee752da7f949a51" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" is_pre_election: true
I20260925 15:27:49.106052 15901 raft_consensus.cc:686] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 LEADER]: Becoming Leader. State: Replica: 58544f8abe8e4e6b9bfe4389541bccb9, State: Running, Role: LEADER
I20260925 15:27:49.106235 15901 consensus_queue.cc:237] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.106948 15630 leader_election.cc:302] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84cedc26793d4a119bd1c0317720933a; no voters: 58544f8abe8e4e6b9bfe4389541bccb9, b3f8fce3e2574c09bcb8a46c70c23f70
I20260925 15:27:49.107116 15902 raft_consensus.cc:2726] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:27:49.107096 15418 catalog_manager.cc:5495] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65). New cstate: current_term: 1 leader_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } health_report { overall_health: UNKNOWN } } }
I20260925 15:27:49.126078 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.123439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.1.163.65:35539)
0925 15:27:49.123523 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.123525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.123532 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:27:49.123571 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.123572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.123591 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.123675 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:27:49.123844 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.124341 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.125204 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.125404 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.125493 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.125687 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:27:49.125804 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:27:49.125898 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.125939 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.125960 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:27:49.128165 15831 raft_consensus.cc:1261] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Refusing update from remote peer 58544f8abe8e4e6b9bfe4389541bccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:49.128209 15697 raft_consensus.cc:1261] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Refusing update from remote peer 58544f8abe8e4e6b9bfe4389541bccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:49.128494 15905 consensus_queue.cc:1035] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:49.128648 15901 consensus_queue.cc:1035] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:49.131512 15909 mvcc.cc:205] Tried to move back new op lower bound from 7333273883145580544 to 7333273883059699712. Current Snapshot: MvccSnapshot[applied={T|T < 7333273883145580544}]
I20260925 15:27:49.132184 15910 mvcc.cc:205] Tried to move back new op lower bound from 7333273883145580544 to 7333273883059699712. Current Snapshot: MvccSnapshot[applied={T|T < 7333273883145580544}]
I20260925 15:27:49.133648 15417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:49.133746 15417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:49.134577 15417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Sending DeleteTablet for 3 replicas of tablet 50fef8c63de640509ee752da7f949a51
I20260925 15:27:49.134985 15677 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fef8c63de640509ee752da7f949a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:49 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:37490
I20260925 15:27:49.135020 15543 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fef8c63de640509ee752da7f949a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:49 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51208
I20260925 15:27:49.135222 15918 tablet_replica.cc:330] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: stopping tablet replica
I20260925 15:27:49.135257 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fef8c63de640509ee752da7f949a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:49 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:33860
I20260925 15:27:49.135339 15918 raft_consensus.cc:2229] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:49.135462 15918 raft_consensus.cc:2258] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:49.135517 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15480
I20260925 15:27:49.136719 15918 ts_tablet_manager.cc:1905] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:49.137194 15920 tablet_replica.cc:330] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: stopping tablet replica
I20260925 15:27:49.137318 15920 raft_consensus.cc:2229] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:49.137445 15920 raft_consensus.cc:2258] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:49.137931 15920 ts_tablet_manager.cc:1905] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:49.138877 15918 ts_tablet_manager.cc:1918] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:49.139071 15918 log.cc:1196] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/50fef8c63de640509ee752da7f949a51
I20260925 15:27:49.139603 15918 ts_tablet_manager.cc:1939] T 50fef8c63de640509ee752da7f949a51 P 84cedc26793d4a119bd1c0317720933a: Deleting consensus metadata
I20260925 15:27:49.139946 15920 ts_tablet_manager.cc:1918] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:49.140194 15920 log.cc:1196] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/50fef8c63de640509ee752da7f949a51
I20260925 15:27:49.140441 15404 catalog_manager.cc:4841] TS 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): tablet 50fef8c63de640509ee752da7f949a51 (table test-table [id=426ef36959e4474db7ee66da3855782a]) successfully deleted
I20260925 15:27:49.140484 15920 ts_tablet_manager.cc:1939] T 50fef8c63de640509ee752da7f949a51 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting consensus metadata
I20260925 15:27:49.140950 15404 catalog_manager.cc:4841] TS b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): tablet 50fef8c63de640509ee752da7f949a51 (table test-table [id=426ef36959e4474db7ee66da3855782a]) successfully deleted
W20260925 15:27:49.142021 15402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:35539 (error 108)
I20260925 15:27:49.142215 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15614
I20260925 15:27:49.143132 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.142996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.0.0.1:51224)
0925 15:27:49.143054 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.143090 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:27:49.143275 15402 catalog_manager.cc:4584] TS 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50fef8c63de640509ee752da7f949a51: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35539: connect: Connection refused (error 111)
I20260925 15:27:49.147315 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15748
I20260925 15:27:49.151986 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15386
I20260925 15:27:49.157754 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--webserver_interface=127.1.163.126
--webserver_port=33023
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:49.242640 15922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:49.242875 15922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:49.242900 15922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:49.244143 15922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:49.244191 15922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:49.244205 15922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:49.244218 15922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:49.244231 15922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:49.245874 15922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:49.246150 15922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:49.246408 15922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:49.256395 15931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.256381 15929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.256381 15928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:49.256407 15922 server_base.cc:1034] running on GCE node
I20260925 15:27:49.257102 15922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:49.257481 15922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:49.258672 15922 hybrid_clock.cc:648] HybridClock initialized: now 1790350069258674 us; error 49 us; skew 500 ppm
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:49.271543 15922 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:49.272154 15922 webserver.cc:460] Webserver started at http://127.1.163.126:33023/ using document root <none> and password file <none>
I20260925 15:27:49.272415 15922 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:49.272511 15922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:49.274124 15922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.274957 15938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.275306 15922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.275422 15922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "e25110f5730145da8353888a96572c71"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "7699fb66fc5e614d8e6a0c8f5a7ff34b"
server_key_iv: "938b7c4e3909785f63661559b56d267c"
server_key_version: "encryptionkey@0"
I20260925 15:27:49.275918 15922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:49.295045 15922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:49.297628 15922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:49.297806 15922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:49.302220 15922 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44899
I20260925 15:27:49.302254 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44899 every 8 connection(s)
I20260925 15:27:49.302657 15922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:49.304296 15991 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:27:49.304332 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15922
I20260925 15:27:49.304803 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:35539
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=33595
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:49.305142 15991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap starting.
I20260925 15:27:49.315696 15991 log.cc:824] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:49.318125 15991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:27:49.318488 15991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap complete.
I20260925 15:27:49.321009 15991 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:49.321259 15991 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25110f5730145da8353888a96572c71, State: Initialized, Role: FOLLOWER
I20260925 15:27:49.321444 15991 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:49.321517 15991 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:49.321578 15991 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:49.321627 15991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:27:49.322726 15991 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:49.322831 15991 leader_election.cc:302] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25110f5730145da8353888a96572c71; no voters:
I20260925 15:27:49.323073 15991 leader_election.cc:288] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:27:49.323114 15995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:27:49.323243 15995 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 LEADER]: Becoming Leader. State: Replica: e25110f5730145da8353888a96572c71, State: Running, Role: LEADER
I20260925 15:27:49.323352 15991 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:49.323357 15995 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:49.323693 15995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25110f5730145da8353888a96572c71. Latest consensus state: current_term: 2 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:49.323787 15995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:49.323989 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:49.324059 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:49.324585 15999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:49.325388 15999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:49.325515 15999 catalog_manager.cc:1259] Loaded cluster ID: 78a7e0c17be94d8b862e01287536a727
I20260925 15:27:49.325551 15999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:27:49.326081 15999 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:27:49.330778 15999 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:27:49.333362 15999 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:27:49.333575 15999 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Loaded TSK: 0
I20260925 15:27:49.333778 15999 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:27:49.397878 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:49.398183 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:49.398209 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:49.399505 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:49.399552 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:49.399581 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:27:49.401196 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:35539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=33595
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:49.401451 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:49.401712 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:49.407274 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.407245 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.407248 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:49.407562 15993 server_base.cc:1034] running on GCE node
I20260925 15:27:49.407802 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:49.408073 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:49.409257 15993 hybrid_clock.cc:648] HybridClock initialized: now 1790350069409238 us; error 44 us; skew 500 ppm
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:49.411917 15993 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:27:49.412439 15993 webserver.cc:460] Webserver started at http://127.1.163.65:33595/ using document root <none> and password file <none>
I20260925 15:27:49.412704 15993 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:49.412760 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:49.414163 15993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.414925 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.415126 15993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.415217 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "dcb33d5bc0c5abf3564b479084180662"
server_key_iv: "c5d7a8a6f1a70f53b73b4e19efd01d82"
server_key_version: "encryptionkey@0"
I20260925 15:27:49.415594 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:49.427383 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:49.429777 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:49.430176 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:49.430459 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:49.430975 16035 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:27:49.432964 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:27:49.433037 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:27:49.433085 15993 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:27:49.433723 15993 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:27:49.433763 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:49.433861 16035 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap starting.
I20260925 15:27:49.441134 15993 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35539
I20260925 15:27:49.441541 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:27:49.445372 16035 log.cc:824] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:49.445670 16142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35539 every 8 connection(s)
I20260925 15:27:49.448453 16035 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:27:49.448801 16035 tablet_bootstrap.cc:492] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap complete.
I20260925 15:27:49.450022 16035 ts_tablet_manager.cc:1397] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20260925 15:27:49.450990 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 15993
I20260925 15:27:49.451313 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:34579
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=33283
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:49.451325 16035 raft_consensus.cc:348] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.451628 16035 raft_consensus.cc:729] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58544f8abe8e4e6b9bfe4389541bccb9, State: Initialized, Role: FOLLOWER
I20260925 15:27:49.451774 16035 consensus_queue.cc:260] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } } peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } }
I20260925 15:27:49.452052 16035 ts_tablet_manager.cc:1428] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:49.457974 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.442309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:58543)
0925 15:27:49.442533 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.442582 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.443110 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.446402 (+ 3292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.446407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.446697 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:49.446883 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.446889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.448074 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.448075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.449047 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.449051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.449678 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.449679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.449815 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.450441 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:49.450450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:49.453645 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:49.456135 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.456144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.456147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.456387 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.456739 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.456740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.456741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.456865 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.457295 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:49.457302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:49.457448 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.457570 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.457765 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":114,"threads_started":1}
I20260925 15:27:49.457985 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.442394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:58543 (local address 127.1.163.126:44899)
0925 15:27:49.445884 (+ 3490us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.445905 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.445959 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:27:49.446110 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.446118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.446183 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.446365 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:49.447018 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.447950 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.449151 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.449570 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.449701 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.453809 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:49.453815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:49.453815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:49.453830 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:49.456024 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.456497 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.456501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.456501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.456666 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.456957 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.456958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.456958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.457216 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:49.457317 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.457629 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.457765 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":109,"threads_started":1}
I20260925 15:27:49.458698 16143 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:49.458827 16143 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:49.459025 16143 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:49.460233 15955 ts_manager.cc:194] Registered new tserver with Master: 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539)
I20260925 15:27:49.461642 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:58543
I20260925 15:27:49.462527 16143 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
I20260925 15:27:49.468151 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.461406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.1.163.65:35539)
0925 15:27:49.464696 (+ 3290us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.464700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.464717 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:27:49.464761 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.464762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.464778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.464913 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.465121 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.465892 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.467155 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.467659 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.467765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.467826 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.468026 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.468072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3243,"thread_start_us":85,"threads_started":1}
I20260925 15:27:49.468150 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.461082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.0.0.1:51232)
0925 15:27:49.461441 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.461477 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.461543 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.464927 (+ 3384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.464931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.464934 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.465012 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.465014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.465986 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.465987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.467025 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.467028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.467770 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.467771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.467857 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.467871 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.468007 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.468079 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":108,"threads_started":1}
I20260925 15:27:49.468914 16077 tablet_service.cc:1514] Processing DeleteTablet for tablet 50fef8c63de640509ee752da7f949a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:49 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51232
I20260925 15:27:49.469555 16152 tablet_replica.cc:330] stopping tablet replica
I20260925 15:27:49.469695 16152 raft_consensus.cc:2229] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:49.469781 16152 raft_consensus.cc:2258] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:49.470103 16152 ts_tablet_manager.cc:1905] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:49.472518 16152 ts_tablet_manager.cc:1918] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:49.472649 16152 log.cc:1196] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/50fef8c63de640509ee752da7f949a51
I20260925 15:27:49.472903 16152 ts_tablet_manager.cc:1939] T 50fef8c63de640509ee752da7f949a51 P 58544f8abe8e4e6b9bfe4389541bccb9: Deleting consensus metadata
I20260925 15:27:49.473490 15939 catalog_manager.cc:4841] TS 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539): tablet 50fef8c63de640509ee752da7f949a51 (table test-table [id=426ef36959e4474db7ee66da3855782a]) successfully deleted
W20260925 15:27:49.543076 16149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:49.543303 16149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:49.543329 16149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:49.544565 16149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:49.544613 16149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:49.544641 16149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:27:49.546142 16149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:34579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=33283
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:49.546416 16149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:49.546676 16149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:49.552111 16158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.552119 16160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:49.552227 16149 server_base.cc:1034] running on GCE node
W20260925 15:27:49.552129 16157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:49.552620 16149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:49.552935 16149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:49.554100 16149 hybrid_clock.cc:648] HybridClock initialized: now 1790350069554067 us; error 48 us; skew 500 ppm
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:49.556730 16149 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:27:49.557257 16149 webserver.cc:460] Webserver started at http://127.1.163.66:33283/ using document root <none> and password file <none>
I20260925 15:27:49.557545 16149 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:49.557603 16149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:49.559027 16149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:27:49.559773 16167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.560096 16149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.560209 16149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "84cedc26793d4a119bd1c0317720933a"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "d5f6122617a32a3b0008bfc8031933c1"
server_key_iv: "2aa63a26c44a9d87504a9de94e4a09f2"
server_key_version: "encryptionkey@0"
I20260925 15:27:49.560709 16149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:49.581836 16149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:49.584160 16149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:49.584348 16149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:49.584669 16149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:49.585075 16149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:49.585135 16149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.585178 16149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:49.585206 16149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.591207 16149 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34579
I20260925 15:27:49.591284 16280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34579 every 8 connection(s)
I20260925 15:27:49.591648 16149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:27:49.598044 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16149
I20260925 15:27:49.598374 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:43173
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=43069
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:49.603150 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.592421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:54767 (local address 127.1.163.126:44899)
0925 15:27:49.592488 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.592490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.592674 (+ 184us) server_negotiation.cc:407] Connection header received
0925 15:27:49.593266 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.593268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.593285 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.593358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:49.594381 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.594948 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.596116 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.596338 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.596396 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.599265 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:49.599267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:49.599267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:49.599276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:49.601187 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.601744 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.601746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.601746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.601821 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.602251 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.602253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.602253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.602438 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:49.602529 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.602873 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.602993 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:27:49.603161 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.592365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:54767)
0925 15:27:49.592532 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.592592 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.593099 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.593437 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.593440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.594013 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:49.594212 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.594217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.595071 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.595072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.596016 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.596019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.596436 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.596437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.596584 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.597184 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:49.597192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:49.599039 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:49.601325 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.601332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.601334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.601643 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.601924 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.601925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.601925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.602059 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.602521 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:49.602526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:49.602669 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.602764 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.602910 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260925 15:27:49.604041 16281 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:49.604138 16281 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:49.604367 16281 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:49.604899 15955 ts_manager.cc:194] Registered new tserver with Master: 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
I20260925 15:27:49.605522 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:54767
W20260925 15:27:49.690662 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:49.690894 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:49.690919 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:49.692098 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:49.692142 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:49.692173 16284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:27:49.693733 16284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:43173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=43069
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:49.693953 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:49.694139 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:49.699256 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.699239 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:49.699231 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:49.699555 16284 server_base.cc:1034] running on GCE node
I20260925 15:27:49.699769 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:49.700016 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:49.701189 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790350069701177 us; error 44 us; skew 500 ppm
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:49.703919 16284 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:27:49.704402 16284 webserver.cc:460] Webserver started at http://127.1.163.67:43069/ using document root <none> and password file <none>
I20260925 15:27:49.704702 16284 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:49.704800 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:49.706173 16284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.706789 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.707008 16284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:49.707108 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "ab9529b351d9764573a0ffdadc1e7d52"
server_key_iv: "4d20031d53cb546443ade617baf899d0"
server_key_version: "encryptionkey@0"
I20260925 15:27:49.707480 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:49.720933 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:49.723547 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:49.723742 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:49.724125 16284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:49.724465 16284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:49.724520 16284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.724576 16284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:49.724603 16284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.730434 16284 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:43173
I20260925 15:27:49.730475 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:43173 every 8 connection(s)
I20260925 15:27:49.730841 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:27:49.735100 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16284
Sep 25 15:27:49 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350069, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:27:49.742139 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.731557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:45491 (local address 127.1.163.126:44899)
0925 15:27:49.731635 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.731638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.731900 (+ 262us) server_negotiation.cc:407] Connection header received
0925 15:27:49.732476 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.732477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.732490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.732663 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:49.733709 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.734285 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.735402 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.735681 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.735767 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.738599 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:49.738600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:49.738601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:49.738610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:49.740454 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.740994 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.740996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.740996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.741057 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.741386 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.741387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.741388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.741531 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:49.741631 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.741891 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.742014 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:27:49.742221 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.731596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:45491)
0925 15:27:49.731775 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.731812 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.732345 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.732744 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.732748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.733379 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:49.733589 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.733594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.734395 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.734396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.735267 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.735270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.735800 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.735801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.735967 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.736776 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:49.736792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:49.738513 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:49.740586 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.740594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.740597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.740918 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.741146 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:49.741147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:49.741147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:49.741288 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:49.741628 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:49.741633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:49.741756 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.741844 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.741968 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 15:27:49.742689 16413 heartbeater.cc:347] Connected to a master server at 127.1.163.126:44899
I20260925 15:27:49.742818 16413 heartbeater.cc:464] Registering TS with master...
I20260925 15:27:49.743026 16413 heartbeater.cc:511] Master 127.1.163.126:44899 requested a full tablet report, sending...
I20260925 15:27:49.743345 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.735589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.1.163.126:44899)
0925 15:27:49.735699 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.735701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.735755 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:27:49.735873 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.735874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.735886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.735958 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:49.736720 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.737517 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.738374 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.738629 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.738711 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.740185 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:49.740187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:49.740187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:49.740196 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:49.742019 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.742412 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.742413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.742413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.742495 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:49.742741 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:49.742742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:49.742742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:49.742869 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:49.743025 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.743120 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.743218 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260925 15:27:49.743448 15955 ts_manager.cc:194] Registered new tserver with Master: b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173)
I20260925 15:27:49.743919 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:45491
I20260925 15:27:49.744186 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:27:49.747102 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.744570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.1.163.126:44899)
0925 15:27:49.744605 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.744607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.744903 (+ 296us) server_negotiation.cc:407] Connection header received
0925 15:27:49.744954 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.744955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.744984 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.745043 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:27:49.745119 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.745579 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.746400 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.746609 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.746675 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.746839 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:27:49.746934 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:27:49.746999 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.747032 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.747052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:27:49.747432 15954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:27:49.749275 15954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:27:49.751488 16077 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 (DEFAULT_TABLE table=test-table [id=a182651aa35348df807dd12fc7ff4454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:49.751716 16077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda0a7e372f437e8635ed726e7033b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:49.753403 16035 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap starting.
I20260925 15:27:49.753746 16035 tablet_bootstrap.cc:654] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:49.754249 16035 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: No bootstrap required, opened a new log
I20260925 15:27:49.754283 16035 ts_tablet_manager.cc:1397] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:49.754365 16035 raft_consensus.cc:348] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.754416 16035 raft_consensus.cc:374] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:49.754446 16035 raft_consensus.cc:729] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58544f8abe8e4e6b9bfe4389541bccb9, State: Initialized, Role: FOLLOWER
I20260925 15:27:49.754508 16035 consensus_queue.cc:260] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.754618 16035 ts_tablet_manager.cc:1428] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:49.755940 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.751387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34579 (local address 127.0.0.1:37494)
0925 15:27:49.751599 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.751609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.751787 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.752495 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.752496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.752498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.752568 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.752570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.753591 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.753591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.754684 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.754687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.755655 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.755656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.755723 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.755728 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.755795 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.755860 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":28,"thread_start_us":66,"threads_started":1}
I20260925 15:27:49.755952 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.751575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.1.163.66:34579)
0925 15:27:49.752171 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.752175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.752208 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:27:49.752247 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.752249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.752338 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.752645 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.752708 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.753487 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.754775 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.755591 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.755656 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.755714 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.755808 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.755861 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":236,"threads_started":1}
I20260925 15:27:49.756127 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.751281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.0.0.1:33864)
0925 15:27:49.751350 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:49.751362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:49.751411 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:49.752089 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:49.752091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:49.752093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:49.752133 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.752134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.753513 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.753513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.754684 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:49.754687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.755885 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.755886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:49.755951 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.755955 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:27:49.756013 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:27:49.756062 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":4}
I20260925 15:27:49.756356 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:49.751333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.1.163.67:43173)
0925 15:27:49.751924 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:49.751929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:49.751940 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:27:49.751983 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:49.751985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:49.751998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:49.752064 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:49.752228 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.753398 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.755035 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:49.755808 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:49.755873 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:49.755928 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:49.756235 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:27:49.756285 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":97,"threads_started":1}
I20260925 15:27:49.756798 16215 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 (DEFAULT_TABLE table=test-table [id=a182651aa35348df807dd12fc7ff4454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:49.757047 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 (DEFAULT_TABLE table=test-table [id=a182651aa35348df807dd12fc7ff4454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:27:49.757164 16215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda0a7e372f437e8635ed726e7033b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:49.757369 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cda0a7e372f437e8635ed726e7033b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:27:49.760919 16423 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Bootstrap starting.
I20260925 15:27:49.761648 16423 tablet_bootstrap.cc:654] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:49.762145 16423 log.cc:824] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:49.762461 16424 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Bootstrap starting.
I20260925 15:27:49.763096 16424 tablet_bootstrap.cc:654] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Neither blocks nor log segments found. Creating new log.
I20260925 15:27:49.763144 16423 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: No bootstrap required, opened a new log
I20260925 15:27:49.763203 16423 ts_tablet_manager.cc:1397] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:27:49.763723 16424 log.cc:824] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:49.764557 16424 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: No bootstrap required, opened a new log
I20260925 15:27:49.764633 16424 ts_tablet_manager.cc:1397] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:27:49.765453 16423 raft_consensus.cc:348] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.765654 16423 raft_consensus.cc:374] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:49.765693 16423 raft_consensus.cc:729] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cedc26793d4a119bd1c0317720933a, State: Initialized, Role: FOLLOWER
I20260925 15:27:49.765813 16423 consensus_queue.cc:260] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.766115 16281 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
I20260925 15:27:49.766113 16423 ts_tablet_manager.cc:1428] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:27:49.766719 16424 raft_consensus.cc:348] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.766909 16424 raft_consensus.cc:374] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:27:49.766955 16424 raft_consensus.cc:729] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f8fce3e2574c09bcb8a46c70c23f70, State: Initialized, Role: FOLLOWER
I20260925 15:27:49.767067 16424 consensus_queue.cc:260] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:49.767357 16413 heartbeater.cc:503] Master 127.1.163.126:44899 was elected leader, sending a full tablet report...
I20260925 15:27:49.767369 16424 ts_tablet_manager.cc:1428] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 15:27:49.842270 16282 tablet.cc:2369] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:27:49.946228 16144 tablet.cc:2369] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:27:49.981422 16414 tablet.cc:2369] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:27:49.999644 16427 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:49.999784 16427 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:50.000156 16427 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539)
I20260925 15:27:50.003407 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.000564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:56681 (local address 127.1.163.67:43173)
0925 15:27:50.000612 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:50.000614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:50.000729 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:27:50.000825 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:50.000827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:50.000843 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:50.000903 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:50.001077 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.001545 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.002592 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.002984 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.003074 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:50.003084 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:50.003243 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:27:50.003286 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:27:50.003420 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.000410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.1.163.66:56681)
0925 15:27:50.000634 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.000644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:50.000747 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:50.000924 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:50.000925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:50.000927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:50.001002 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:50.001003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.001648 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.001649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:50.002488 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:50.002490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.003072 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.003073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:50.003150 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:50.003159 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:27:50.003234 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:27:50.003279 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260925 15:27:50.003748 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.000386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.1.163.66:47991)
0925 15:27:50.000442 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.000455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:50.000509 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:50.000803 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:50.000804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:50.000806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:27:50.000930 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:50.000933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.001612 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.001613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:50.002724 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:50.002727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.003447 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.003449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:50.003555 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:50.003559 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:27:50.003610 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:27:50.003673 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":98}
I20260925 15:27:50.003820 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.000508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:47991 (local address 127.1.163.65:35539)
0925 15:27:50.000655 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:50.000657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:50.000674 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:27:50.000701 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:50.000702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:50.000711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:50.000798 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:27:50.001012 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.001522 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.002895 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.003352 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.003449 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:50.003459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:50.003682 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:27:50.003736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 15:27:50.003809 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda0a7e372f437e8635ed726e7033b0" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" is_pre_election: true
I20260925 15:27:50.003996 16367 raft_consensus.cc:2454] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cedc26793d4a119bd1c0317720933a in term 0.
I20260925 15:27:50.004139 16097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda0a7e372f437e8635ed726e7033b0" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" is_pre_election: true
I20260925 15:27:50.004307 16097 raft_consensus.cc:2454] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cedc26793d4a119bd1c0317720933a in term 0.
I20260925 15:27:50.004310 16170 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70; no voters:
I20260925 15:27:50.004490 16427 raft_consensus.cc:2781] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:27:50.004580 16427 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:27:50.004616 16427 raft_consensus.cc:3037] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:50.005348 16427 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:50.005497 16427 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 election: Requested vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539)
I20260925 15:27:50.005681 16097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda0a7e372f437e8635ed726e7033b0" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
I20260925 15:27:50.005719 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cda0a7e372f437e8635ed726e7033b0" candidate_uuid: "84cedc26793d4a119bd1c0317720933a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
I20260925 15:27:50.005795 16367 raft_consensus.cc:3037] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:50.005788 16097 raft_consensus.cc:3037] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:27:50.006699 16097 raft_consensus.cc:2454] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cedc26793d4a119bd1c0317720933a in term 1.
I20260925 15:27:50.006762 16367 raft_consensus.cc:2454] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cedc26793d4a119bd1c0317720933a in term 1.
I20260925 15:27:50.006950 16170 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70; no voters:
I20260925 15:27:50.007064 16427 raft_consensus.cc:2781] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:27:50.007257 16427 raft_consensus.cc:686] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 1 LEADER]: Becoming Leader. State: Replica: 84cedc26793d4a119bd1c0317720933a, State: Running, Role: LEADER
I20260925 15:27:50.007340 16427 consensus_queue.cc:237] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:50.008483 15955 catalog_manager.cc:5495] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cedc26793d4a119bd1c0317720933a (127.1.163.66). New cstate: current_term: 1 leader_uuid: "84cedc26793d4a119bd1c0317720933a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } health_report { overall_health: UNKNOWN } } }
I20260925 15:27:50.044914 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.042463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.1.163.66:34579)
0925 15:27:50.042546 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:50.042548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:50.042554 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:27:50.042591 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:50.042592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:50.042609 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:50.042688 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:27:50.042803 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.043306 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.044114 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:50.044307 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:50.044393 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:50.044525 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:27:50.044653 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:27:50.044751 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:50.044777 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:27:50.044798 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:27:50.046795 16097 raft_consensus.cc:1261] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Refusing update from remote peer 84cedc26793d4a119bd1c0317720933a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:50.046846 16367 raft_consensus.cc:1261] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Refusing update from remote peer 84cedc26793d4a119bd1c0317720933a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:27:50.047199 16427 consensus_queue.cc:1035] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:50.047407 16431 consensus_queue.cc:1035] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:27:50.050149 16436 mvcc.cc:205] Tried to move back new op lower bound from 7333273886908801024 to 7333273886750670848. Current Snapshot: MvccSnapshot[applied={T|T < 7333273886908801024}]
I20260925 15:27:50.050184 16435 mvcc.cc:205] Tried to move back new op lower bound from 7333273886908801024 to 7333273886750670848. Current Snapshot: MvccSnapshot[applied={T|T < 7333273886908801024}]
I20260925 15:27:50.052030 15955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:50.052145 15955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:27:50.052928 15955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Sending DeleteTablet for 3 replicas of tablet 4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:50.053260 16077 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51232
I20260925 15:27:50.053634 16443 tablet_replica.cc:330] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: stopping tablet replica
I20260925 15:27:50.053637 16215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:37494
I20260925 15:27:50.053710 16443 raft_consensus.cc:2229] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:50.053776 16443 raft_consensus.cc:2258] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:50.053954 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15993
I20260925 15:27:50.053927 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cda0a7e372f437e8635ed726e7033b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:27:50 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:33864
I20260925 15:27:50.054709 16445 tablet_replica.cc:330] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: stopping tablet replica
I20260925 15:27:50.054832 16445 raft_consensus.cc:2229] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:27:50.054994 16445 raft_consensus.cc:2258] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:50.055002 16444 tablet_replica.cc:330] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: stopping tablet replica
I20260925 15:27:50.055164 16444 raft_consensus.cc:2229] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:27:50.055258 16444 raft_consensus.cc:2258] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:27:50.055809 16444 ts_tablet_manager.cc:1905] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:50.055809 16445 ts_tablet_manager.cc:1905] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:27:50.058115 16444 ts_tablet_manager.cc:1918] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:50.058115 16445 ts_tablet_manager.cc:1918] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:27:50.058204 16445 log.cc:1196] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:50.058204 16444 log.cc:1196] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:50.058526 16445 ts_tablet_manager.cc:1939] T 4cda0a7e372f437e8635ed726e7033b0 P b3f8fce3e2574c09bcb8a46c70c23f70: Deleting consensus metadata
I20260925 15:27:50.058526 16444 ts_tablet_manager.cc:1939] T 4cda0a7e372f437e8635ed726e7033b0 P 84cedc26793d4a119bd1c0317720933a: Deleting consensus metadata
I20260925 15:27:50.059115 15941 catalog_manager.cc:4841] TS 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): tablet 4cda0a7e372f437e8635ed726e7033b0 (table test-table [id=a182651aa35348df807dd12fc7ff4454]) successfully deleted
I20260925 15:27:50.059231 15941 catalog_manager.cc:4841] TS b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): tablet 4cda0a7e372f437e8635ed726e7033b0 (table test-table [id=a182651aa35348df807dd12fc7ff4454]) successfully deleted
W20260925 15:27:50.062084 15939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:35539 (error 108)
I20260925 15:27:50.062497 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 16149
I20260925 15:27:50.062675 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.062508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35539 (local address 127.0.0.1:51238)
0925 15:27:50.062579 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.062632 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:27:50.063177 15939 catalog_manager.cc:4584] TS 58544f8abe8e4e6b9bfe4389541bccb9 (127.1.163.65:35539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cda0a7e372f437e8635ed726e7033b0: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35539: connect: Connection refused (error 111)
I20260925 15:27:50.067422 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 16284
I20260925 15:27:50.072165 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 15922
I20260925 15:27:50.077976 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--webserver_interface=127.1.163.126
--webserver_port=33023
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:27:50.163441 16447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:50.163652 16447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:50.163677 16447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:50.164994 16447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:27:50.165040 16447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:50.165055 16447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:50.165068 16447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:27:50.165081 16447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:27:50.166658 16447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:50.166970 16447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:50.167232 16447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:50.172256 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.172252 16455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.172271 16447 server_base.cc:1034] running on GCE node
W20260925 15:27:50.172252 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.172847 16447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:50.173183 16447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:50.174358 16447 hybrid_clock.cc:648] HybridClock initialized: now 1790350070174339 us; error 37 us; skew 500 ppm
Sep 25 15:27:50 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:50.177136 16447 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:27:50.177636 16447 webserver.cc:460] Webserver started at http://127.1.163.126:33023/ using document root <none> and password file <none>
I20260925 15:27:50.177903 16447 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:50.177974 16447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:50.179349 16447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:50.180130 16462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.180338 16447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:50.180466 16447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "e25110f5730145da8353888a96572c71"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "7699fb66fc5e614d8e6a0c8f5a7ff34b"
server_key_iv: "938b7c4e3909785f63661559b56d267c"
server_key_version: "encryptionkey@0"
I20260925 15:27:50.180889 16447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:50.193082 16447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:50.195312 16447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:50.195485 16447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:50.199821 16447 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44899
I20260925 15:27:50.199976 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44899 every 8 connection(s)
I20260925 15:27:50.200305 16447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:27:50.202034 16515 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:27:50.202737 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap starting.
I20260925 15:27:50.203784 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16447
I20260925 15:27:50.204118 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:35539
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=33595
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:50.211328 16515 log.cc:824] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:50.213747 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:27:50.214080 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71: Bootstrap complete.
I20260925 15:27:50.216925 16515 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:50.217212 16515 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25110f5730145da8353888a96572c71, State: Initialized, Role: FOLLOWER
I20260925 15:27:50.217447 16515 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:50.217545 16515 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:27:50.217607 16515 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:27:50.217660 16515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:27:50.218861 16515 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:50.218997 16515 leader_election.cc:302] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25110f5730145da8353888a96572c71; no voters:
I20260925 15:27:50.219219 16515 leader_election.cc:288] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:27:50.219261 16519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:27:50.219391 16519 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [term 3 LEADER]: Becoming Leader. State: Replica: e25110f5730145da8353888a96572c71, State: Running, Role: LEADER
I20260925 15:27:50.219491 16519 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } }
I20260925 15:27:50.219545 16515 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:27:50.219775 16519 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25110f5730145da8353888a96572c71. Latest consensus state: current_term: 3 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:50.219853 16519 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:50.219779 16520 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e25110f5730145da8353888a96572c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25110f5730145da8353888a96572c71" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44899 } } }
I20260925 15:27:50.220016 16520 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25110f5730145da8353888a96572c71 [sys.catalog]: This master's current role is: LEADER
I20260925 15:27:50.220064 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:27:50.221052 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:27:50.221186 16523 catalog_manager.cc:1259] Loaded cluster ID: 78a7e0c17be94d8b862e01287536a727
I20260925 15:27:50.221236 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:27:50.221751 16523 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb18e9c9ad google::LogMessage::Fail() at ??:0
@ 0x7fdb18e9f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb18e9c4de google::LogMessage::Flush() at ??:0
@ 0x7fdb18e9da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb20dbd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdb20dc68b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb1ad6a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb1ad5f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb1bbc36db start_thread at ??:0
@ 0x7fdb1799a61f clone at ??:0
W20260925 15:27:50.332032 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:50.332262 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:50.332289 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:50.333706 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:50.333758 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:50.333791 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:27:50.335472 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:35539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=33595
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:50.335738 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:50.336011 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:50.341687 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.341746 16517 server_base.cc:1034] running on GCE node
W20260925 15:27:50.341691 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.341919 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.342129 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:50.342440 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:50.343606 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790350070343574 us; error 48 us; skew 500 ppm
Sep 25 15:27:50 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:50.347077 16517 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:27:50.347550 16517 webserver.cc:460] Webserver started at http://127.1.163.65:33595/ using document root <none> and password file <none>
I20260925 15:27:50.347792 16517 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:50.347841 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:50.349423 16517 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:27:50.350333 16561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.350582 16517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:27:50.350678 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "58544f8abe8e4e6b9bfe4389541bccb9"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "dcb33d5bc0c5abf3564b479084180662"
server_key_iv: "c5d7a8a6f1a70f53b73b4e19efd01d82"
server_key_version: "encryptionkey@0"
I20260925 15:27:50.351181 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:50.381026 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:50.384068 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:50.384225 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:50.384493 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:50.385062 16568 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:27:50.386590 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:27:50.386649 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:27:50.386683 16517 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:27:50.387624 16517 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:27:50.387670 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:50.387784 16568 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap starting.
I20260925 15:27:50.395218 16517 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35539
I20260925 15:27:50.395277 16675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35539 every 8 connection(s)
I20260925 15:27:50.395684 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:27:50.398631 16568 log.cc:824] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:27:50.400504 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16517
I20260925 15:27:50.400818 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:34579
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=33283
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:50.400933 16568 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:27:50.401204 16568 tablet_bootstrap.cc:492] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Bootstrap complete.
I20260925 15:27:50.402381 16568 ts_tablet_manager.cc:1397] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260925 15:27:50.403421 16568 raft_consensus.cc:348] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:50.403644 16568 raft_consensus.cc:729] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58544f8abe8e4e6b9bfe4389541bccb9, State: Initialized, Role: FOLLOWER
I20260925 15:27:50.403784 16568 consensus_queue.cc:260] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:50.404088 16568 ts_tablet_manager.cc:1428] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260925 15:27:50.486153 16680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:50.486411 16680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:50.486455 16680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:50.487701 16680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:50.487774 16680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:50.487833 16680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:27:50.490836 16680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:34579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=33283
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:50.491137 16680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:50.491432 16680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:50.493804 16678 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:27:50.396358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:52297)
0925 15:27:50.396553 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.396604 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:50.397368 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:50.493583 (+ 96215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260925 15:27:50.494019 16676 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:44899 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:44899: Client connection negotiation failed: client connection to 127.1.163.126:44899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:27:50.494395 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.494269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:48507)
0925 15:27:50.494334 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.494374 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:27:50.494719 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.494623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:35579)
0925 15:27:50.494672 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.494696 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:27:50.494831 16676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:27:50.497453 16689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.497468 16687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.497467 16686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.497731 16680 server_base.cc:1034] running on GCE node
I20260925 15:27:50.497954 16680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:50.498229 16680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:50.499436 16680 hybrid_clock.cc:648] HybridClock initialized: now 1790350070499409 us; error 37 us; skew 500 ppm
Sep 25 15:27:50 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:50.502211 16680 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:27:50.502663 16680 webserver.cc:460] Webserver started at http://127.1.163.66:33283/ using document root <none> and password file <none>
I20260925 15:27:50.502920 16680 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:50.502995 16680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:50.504396 16680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:27:50.505131 16696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.505344 16680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:27:50.505450 16680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "84cedc26793d4a119bd1c0317720933a"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "d5f6122617a32a3b0008bfc8031933c1"
server_key_iv: "2aa63a26c44a9d87504a9de94e4a09f2"
server_key_version: "encryptionkey@0"
I20260925 15:27:50.505810 16680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:50.516734 16680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:50.519220 16680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:50.519399 16680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:50.519639 16680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:50.519954 16680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:50.520009 16680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.520052 16680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:50.520085 16680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.526191 16680 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:34579
I20260925 15:27:50.526234 16809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:34579 every 8 connection(s)
I20260925 15:27:50.526634 16680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:27:50.527436 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16680
I20260925 15:27:50.527740 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:43173
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:44899
--webserver_port=43069
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:27:50.527952 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.527500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:56949)
0925 15:27:50.527666 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.527737 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
W20260925 15:27:50.528373 16810 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:44899 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:44899: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
I20260925 15:27:50.528688 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.528604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:52357)
0925 15:27:50.528637 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.528669 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 15:27:50.529016 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.528924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:46625)
0925 15:27:50.528972 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.528996 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:27:50.529137 16810 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:27:50.611368 16813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:27:50.611590 16813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:27:50.611615 16813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:27:50.612867 16813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:27:50.612911 16813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:27:50.612938 16813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:27:50.614393 16813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:43173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=43069
--tserver_master_addrs=127.1.163.126:44899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:27:50.614590 16813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:27:50.614775 16813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:27:50.620024 16821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.620052 16819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:27:50.620010 16818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:27:50.620330 16813 server_base.cc:1034] running on GCE node
I20260925 15:27:50.620612 16813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:27:50.620883 16813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:27:50.622084 16813 hybrid_clock.cc:648] HybridClock initialized: now 1790350070622062 us; error 33 us; skew 500 ppm
Sep 25 15:27:50 dist-test-slave-hvv5 krb5kdc[15373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:27:50.625890 16813 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:27:50.626698 16813 webserver.cc:460] Webserver started at http://127.1.163.67:43069/ using document root <none> and password file <none>
I20260925 15:27:50.627130 16813 fs_manager.cc:359] Metadata directory not provided
I20260925 15:27:50.627200 16813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:27:50.628938 16813 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:27:50.629659 16828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.629819 16813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:27:50.629935 16813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b3f8fce3e2574c09bcb8a46c70c23f70"
format_stamp: "Formatted at 2026-09-25 15:27:48 on dist-test-slave-hvv5"
server_key: "ab9529b351d9764573a0ffdadc1e7d52"
server_key_iv: "4d20031d53cb546443ade617baf899d0"
server_key_version: "encryptionkey@0"
I20260925 15:27:50.630314 16813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:27:50.641012 16813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:27:50.643380 16813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:27:50.643533 16813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:27:50.643827 16813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:27:50.644186 16813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:27:50.644222 16813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.644248 16813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:27:50.644295 16813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:27:50.651839 16813 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:43173
I20260925 15:27:50.651991 16941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:43173 every 8 connection(s)
I20260925 15:27:50.652309 16813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:27:50.653239 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 16813
I20260925 15:27:50.653627 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.653079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:41675)
0925 15:27:50.653302 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.653407 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
W20260925 15:27:50.654174 16942 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:44899 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:44899: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
I20260925 15:27:50.654537 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.654414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:46823)
0925 15:27:50.654469 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.654505 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:27:50.654894 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:50.654777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:51549)
0925 15:27:50.654829 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:50.654860 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:27:50.655071 16942 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:27:51.495409 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.495193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:51437)
0925 15:27:51.495321 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:51.495362 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 15:27:51.529763 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.529511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:42157)
0925 15:27:51.529667 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:51.529702 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 15:27:51.655719 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.655466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:37539)
0925 15:27:51.655631 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:51.655667 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260925 15:27:51.670775 16949 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:51.670977 16949 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:51.671319 16949 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
Sep 25 15:27:51 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.67@KRBTEST.COM
Sep 25 15:27:51 dist-test-slave-hvv5 krb5kdc[15373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350070, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.66@KRBTEST.COM
I20260925 15:27:51.682154 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.671694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:34579 (local address 127.1.163.65:34535)
0925 15:27:51.671800 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:51.671809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:51.671857 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:51.672727 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:51.672729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:51.673100 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:51.673328 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:51.673335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.674511 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.674512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:51.675400 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:51.675402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.676025 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.676026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:51.676147 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:51.676764 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:51.676773 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:51.678622 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:51.680579 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:51.680580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:51.680581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:51.680804 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:51.681077 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:51.681078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:51.681078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:51.681143 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:51.681701 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:51.681702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:51.681754 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:27:51.681845 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:27:51.682042 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":77,"thread_start_us":53,"threads_started":1}
I20260925 15:27:51.682173 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.671669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:47023 (local address 127.1.163.67:43173)
0925 15:27:51.671854 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:51.671872 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:51.671945 (+ 73us) server_negotiation.cc:407] Connection header received
0925 15:27:51.672416 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:51.672419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:51.672462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:51.672669 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:51.673479 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.674402 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.675487 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.675912 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.676048 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:51.678388 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:51.678393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:51.678393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:51.678406 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:51.680316 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:51.680837 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:51.680841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:51.680841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:51.680955 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:51.681214 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:51.681215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:51.681216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:51.681469 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:51.681535 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:51.681912 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 15:27:51.682044 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260925 15:27:51.682185 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.671852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:34535 (local address 127.1.163.66:34579)
0925 15:27:51.671982 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:27:51.671989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:27:51.672014 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:27:51.672426 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:27:51.672428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:27:51.672464 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:27:51.672691 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:27:51.673457 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.674391 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.675501 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.675918 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.676048 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:51.678726 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:27:51.678730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:27:51.678730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:27:51.678742 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:27:51.680443 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:51.680885 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:51.680893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:51.680893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:51.680977 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:27:51.681344 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:27:51.681344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:27:51.681345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:27:51.681619 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:27:51.681706 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:27:51.681909 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:27:51.682059 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 15:27:51.682452 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:51.671565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:43173 (local address 127.1.163.65:47023)
0925 15:27:51.671623 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:51.671635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:27:51.671692 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:27:51.672725 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:27:51.672729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:27:51.673101 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:27:51.673330 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:51.673336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.674527 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.674528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:51.675390 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:27:51.675392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:27:51.676023 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:27:51.676024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:27:51.676145 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:27:51.676829 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:27:51.676833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:27:51.678270 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:27:51.680412 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:51.680420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:51.680422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:51.680751 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:51.681022 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:27:51.681023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:27:51.681024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:27:51.681143 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:27:51.681543 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:27:51.681550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:27:51.681692 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:27:51.681840 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:27:51.682131 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":189}
W20260925 15:27:51.683245 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
W20260925 15:27:51.683314 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:51.683331 16564 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9; no voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70
I20260925 15:27:51.683420 16949 raft_consensus.cc:2726] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:27:52.496078 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:52.495864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:36765)
0925 15:27:52.495987 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:52.496029 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 15:27:52.530694 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:52.530354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:48195)
0925 15:27:52.530598 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:52.530647 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":195,"threads_started":1}
I20260925 15:27:52.656608 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:52.656294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:55419)
0925 15:27:52.656452 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:52.656497 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260925 15:27:53.490337 16956 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:53.490458 16956 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:53.490665 16956 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
W20260925 15:27:53.491091 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
W20260925 15:27:53.491153 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:53.491168 16564 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9; no voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70
I20260925 15:27:53.491243 16956 raft_consensus.cc:2726] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:27:53.496973 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:53.496713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:60033)
0925 15:27:53.496878 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:53.496928 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260925 15:27:53.531558 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:53.531327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:49445)
0925 15:27:53.531475 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:53.531509 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 15:27:53.657421 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:53.657146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:59155)
0925 15:27:53.657328 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:53.657375 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260925 15:27:54.497941 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:54.497659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:58443)
0925 15:27:54.497848 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:54.497894 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":141,"threads_started":1}
I20260925 15:27:54.532332 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:54.532084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:48455)
0925 15:27:54.532231 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:54.532282 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260925 15:27:54.658197 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:54.657941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:37743)
0925 15:27:54.658101 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:54.658151 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260925 15:27:55.498864 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:55.498586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:58851)
0925 15:27:55.498725 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:55.498813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260925 15:27:55.533164 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:55.532944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:42265)
0925 15:27:55.533066 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:55.533112 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260925 15:27:55.659080 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:55.658772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:46089)
0925 15:27:55.658977 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:55.659025 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":128,"threads_started":1}
I20260925 15:27:56.001912 16966 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:27:56.002075 16966 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:27:56.002282 16966 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
W20260925 15:27:56.002691 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
W20260925 15:27:56.002797 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
I20260925 15:27:56.002897 16564 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9; no voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70
I20260925 15:27:56.003041 16966 raft_consensus.cc:2726] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:27:56.499773 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:56.499436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:60421)
0925 15:27:56.499635 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:56.499714 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":113,"threads_started":1}
I20260925 15:27:56.534031 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:56.533758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:55155)
0925 15:27:56.533935 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:56.533966 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260925 15:27:56.659937 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:56.659688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:54157)
0925 15:27:56.659841 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:56.659887 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260925 15:27:57.500597 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:57.500312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:48791)
0925 15:27:57.500461 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:57.500500 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260925 15:27:57.534749 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:57.534533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:60175)
0925 15:27:57.534667 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:57.534703 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260925 15:27:57.660933 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:57.660613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:48967)
0925 15:27:57.660786 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:57.660838 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260925 15:27:58.501278 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:58.501054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:50923)
0925 15:27:58.501192 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:58.501232 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260925 15:27:58.535665 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:58.535312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:44975)
0925 15:27:58.535529 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:58.535599 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":126,"threads_started":1}
I20260925 15:27:58.661895 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:58.661608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:43711)
0925 15:27:58.661776 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:58.661823 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260925 15:27:59.502226 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:59.501958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:36729)
0925 15:27:59.502131 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:59.502180 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260925 15:27:59.536486 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:59.536239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:50767)
0925 15:27:59.536401 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:59.536438 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260925 15:27:59.662829 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:27:59.662583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:34463)
0925 15:27:59.662733 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:27:59.662781 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260925 15:28:00.503058 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:00.502778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:54421)
0925 15:28:00.502957 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:00.503005 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260925 15:28:00.537312 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:00.537050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:58605)
0925 15:28:00.537204 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:00.537251 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 15:28:00.663626 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:00.663397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:58135)
0925 15:28:00.663536 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:00.663577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 15:28:00.972402 16982 raft_consensus.cc:482] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:00.972510 16982 raft_consensus.cc:504] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f8fce3e2574c09bcb8a46c70c23f70" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 43173 } } peers { permanent_uuid: "84cedc26793d4a119bd1c0317720933a" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 34579 } } peers { permanent_uuid: "58544f8abe8e4e6b9bfe4389541bccb9" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35539 } }
I20260925 15:28:00.972746 16982 leader_election.cc:288] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173), 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579)
W20260925 15:28:00.973312 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3f8fce3e2574c09bcb8a46c70c23f70 (127.1.163.67:43173): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
W20260925 15:28:00.973418 16564 leader_election.cc:341] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84cedc26793d4a119bd1c0317720933a (127.1.163.66:34579): Not found: Tablet not found: 4cda0a7e372f437e8635ed726e7033b0
I20260925 15:28:00.973452 16564 leader_election.cc:302] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58544f8abe8e4e6b9bfe4389541bccb9; no voters: 84cedc26793d4a119bd1c0317720933a, b3f8fce3e2574c09bcb8a46c70c23f70
I20260925 15:28:00.973558 16982 raft_consensus.cc:2726] T 4cda0a7e372f437e8635ed726e7033b0 P 58544f8abe8e4e6b9bfe4389541bccb9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:28:01.503892 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:01.503645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:49021)
0925 15:28:01.503794 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:01.503842 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260925 15:28:01.538180 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:01.537866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:34909)
0925 15:28:01.538067 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:01.538124 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":140,"threads_started":1}
I20260925 15:28:01.664402 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:01.664160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:44623)
0925 15:28:01.664307 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:01.664351 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260925 15:28:02.504715 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:02.504388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:51445)
0925 15:28:02.504623 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:02.504671 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":205,"threads_started":1}
I20260925 15:28:02.538959 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:02.538702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:45653)
0925 15:28:02.538830 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:02.538870 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260925 15:28:02.665156 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:02.664922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:40151)
0925 15:28:02.665068 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:02.665107 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260925 15:28:03.505453 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:03.505187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:49495)
0925 15:28:03.505352 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:03.505400 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260925 15:28:03.539695 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:03.539455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:59903)
0925 15:28:03.539597 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:03.539645 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260925 15:28:03.665925 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:03.665673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:39269)
0925 15:28:03.665817 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:03.665869 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 15:28:04.506315 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:04.506055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:35869)
0925 15:28:04.506212 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:04.506260 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260925 15:28:04.540392 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:04.540158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:41243)
0925 15:28:04.540305 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:04.540343 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260925 15:28:04.666846 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:04.666536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.67:51981)
0925 15:28:04.666726 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:04.666761 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260925 15:28:05.507076 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:05.506879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.65:56129)
0925 15:28:05.506999 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:05.507032 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260925 15:28:05.541312 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:05.541072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:44899 (local address 127.1.163.66:43167)
0925 15:28:05.541223 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:05.541266 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260925 15:28:05.654453 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 16517
I20260925 15:28:05.659590 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 16680
I20260925 15:28:05.664032 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 16813
2026-09-25T15:28:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18585 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:28:05 dist-test-slave-hvv5 krb5kdc[16999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:28:05 dist-test-slave-hvv5 krb5kdc[16999](info): set up 2 sockets
Sep 25 15:28:05 dist-test-slave-hvv5 krb5kdc[16999](info): commencing operation
krb5kdc: starting...
W20260925 15:28:06.704844 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:28:06 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:28:06.832795 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:06.916306 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:06.916627 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:06.916682 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:06.917997 17012 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:28:06.918079 17012 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:28:06.918100 17012 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:28:06.918119 17012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:28:06.918149 17012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:06.918180 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:06.918205 17012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:28:06.918238 17012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:28:06.919893 17012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:06.920207 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:06.920460 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:06.926079 17017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:06.926177 17012 server_base.cc:1034] running on GCE node
W20260925 15:28:06.926129 17020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:06.926126 17018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:06.926643 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:06.927026 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:06.928277 17012 hybrid_clock.cc:648] HybridClock initialized: now 1790350086928254 us; error 64 us; skew 500 ppm
Sep 25 15:28:06 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:06.930997 17012 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:28:06.931514 17012 webserver.cc:460] Webserver started at http://127.1.163.126:44675/ using document root <none> and password file <none>
I20260925 15:28:06.931743 17012 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:06.931811 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:06.931972 17012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:28:06.933152 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "1ba9d19c66c74c9496a7b3da228f5406"
format_stamp: "Formatted at 2026-09-25 15:28:06 on dist-test-slave-hvv5"
server_key: "146f9f0ea032a2f5277793edd14b5c8b"
server_key_iv: "ec0080601d7550e88e5ea8714db095dd"
server_key_version: "encryptionkey@0"
I20260925 15:28:06.933555 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "1ba9d19c66c74c9496a7b3da228f5406"
format_stamp: "Formatted at 2026-09-25 15:28:06 on dist-test-slave-hvv5"
server_key: "146f9f0ea032a2f5277793edd14b5c8b"
server_key_iv: "ec0080601d7550e88e5ea8714db095dd"
server_key_version: "encryptionkey@0"
I20260925 15:28:06.935035 17012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:06.935804 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:06.935998 17012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:06.936115 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "1ba9d19c66c74c9496a7b3da228f5406"
format_stamp: "Formatted at 2026-09-25 15:28:06 on dist-test-slave-hvv5"
server_key: "146f9f0ea032a2f5277793edd14b5c8b"
server_key_iv: "ec0080601d7550e88e5ea8714db095dd"
server_key_version: "encryptionkey@0"
I20260925 15:28:06.936194 17012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:06.962304 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:06.964490 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:06.964701 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:06.968823 17012 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36167
I20260925 15:28:06.968860 17079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36167 every 8 connection(s)
I20260925 15:28:06.969262 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:28:06.970151 17080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:06.972437 17080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap starting.
I20260925 15:28:06.973075 17080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:06.973400 17080 log.cc:824] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:06.974112 17080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: No bootstrap required, opened a new log
I20260925 15:28:06.975725 17080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:06.975907 17080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:06.975934 17080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Initialized, Role: FOLLOWER
I20260925 15:28:06.976109 17080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:06.976212 17080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:28:06.976236 17080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:28:06.976315 17080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:06.976899 17080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:06.977010 17080 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba9d19c66c74c9496a7b3da228f5406; no voters:
I20260925 15:28:06.977182 17080 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:28:06.977240 17083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:28:06.977451 17083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Running, Role: LEADER
I20260925 15:28:06.977474 17080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:28:06.977545 17083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:06.978008 17085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba9d19c66c74c9496a7b3da228f5406. Latest consensus state: current_term: 1 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:06.977998 17084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:06.978240 17084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:06.978379 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17012
I20260925 15:28:06.978480 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:28:06.978698 1677 external_mini_cluster.cc:1428] Setting key 3e45b5248a1888df0d5db9c7fb6176a1
I20260925 15:28:06.978145 17085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:06.979614 17091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:28:06.980161 17091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:28:06.982663 17091 catalog_manager.cc:1347] Generated new cluster ID: b7ee3d5b55dc4441b7f56e641299c522
I20260925 15:28:06.982718 17091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:28:06 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:06.989380 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:06.979463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.1.163.126:36167)
0925 15:28:06.979646 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:06.979658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:06.979694 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:28:06.979814 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:06.979821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:06.979865 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:06.980002 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:06.980866 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:06.981878 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:06.982883 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:06.983307 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:06.983426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:06.985921 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:06.985925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:06.985925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:06.985938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:06.987682 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:06.988079 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:06.988083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:06.988083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:06.988176 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:06.988399 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:06.988400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:06.988400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:06.988738 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:06.988887 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:06.989003 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:28:06.989150 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260925 15:28:07.006286 17091 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:28:07.006883 17091 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:28:07.012518 17091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Generated new TSK 0
I20260925 15:28:07.012802 17091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:28:07.048271 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:07.133420 17106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:07.133641 17106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:07.133668 17106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:07.134934 17106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:07.134986 17106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:07.135017 17106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:28:07.136590 17106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:07.136852 17106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:07.137112 17106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:07.142627 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:07.142647 17111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:07.142638 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:07.142828 17106 server_base.cc:1034] running on GCE node
I20260925 15:28:07.143126 17106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:07.143420 17106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:07.144629 17106 hybrid_clock.cc:648] HybridClock initialized: now 1790350087144603 us; error 45 us; skew 500 ppm
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:07.147363 17106 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:28:07.147837 17106 webserver.cc:460] Webserver started at http://127.1.163.65:34083/ using document root <none> and password file <none>
I20260925 15:28:07.148111 17106 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:07.148181 17106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:07.148334 17106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:28:07.149478 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "193bc2deeaf342babe54a2a3edcbe7fa"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f1a430c04de8f5a59d5eca0baaba9426"
server_key_iv: "0cd6f75f72595f41e62f9ecf95c1af3f"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.149857 17106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "193bc2deeaf342babe54a2a3edcbe7fa"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f1a430c04de8f5a59d5eca0baaba9426"
server_key_iv: "0cd6f75f72595f41e62f9ecf95c1af3f"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.151262 17106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:07.152153 17121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.152340 17106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:07.152449 17106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "193bc2deeaf342babe54a2a3edcbe7fa"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f1a430c04de8f5a59d5eca0baaba9426"
server_key_iv: "0cd6f75f72595f41e62f9ecf95c1af3f"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.152525 17106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:07.161945 17106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:07.164261 17106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:07.164469 17106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:07.164778 17106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:07.165143 17106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:07.165202 17106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.165244 17106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:07.165295 17106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.171938 17106 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:39311
I20260925 15:28:07.172050 17234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:39311 every 8 connection(s)
I20260925 15:28:07.172398 17106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:28:07.173535 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17106
I20260925 15:28:07.173668 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:28:07.173914 1677 external_mini_cluster.cc:1428] Setting key db8e1aea67c2df8fb774e0218090be0c
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
I20260925 15:28:07.184623 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.173421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:40427 (local address 127.1.163.126:36167)
0925 15:28:07.173482 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.173485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.173700 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:28:07.174489 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.174491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.174505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.174554 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:07.175931 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.176801 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.177946 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.178294 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.178373 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.181120 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:07.181122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:07.181123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:07.181134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:07.182959 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.183508 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.183509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.183510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.183574 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.183838 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.183838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.183839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.183955 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:07.184012 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.184397 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.184486 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:28:07.184983 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.173301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:40427)
0925 15:28:07.173534 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.173591 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.174373 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.174627 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.174632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.175511 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:07.175812 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.175819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.176935 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.176936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.177857 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.177859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.178391 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.178392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.178546 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.179238 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:07.179255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:07.181008 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:07.183092 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.183101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.183103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.183429 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.183639 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.183640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.183640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.183750 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.184066 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:07.184073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:07.184219 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.184349 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.184644 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260925 15:28:07.185648 17235 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:07.185768 17235 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:07.185989 17235 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
I20260925 15:28:07.186585 17044 ts_manager.cc:194] Registered new tserver with Master: 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65:39311)
I20260925 15:28:07.187197 17044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:40427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:28:07.205262 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:07.289716 17240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:07.289961 17240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:07.289989 17240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:07.291302 17240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:07.291357 17240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:07.291389 17240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:28:07.292997 17240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:07.293277 17240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:07.293527 17240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:07.299074 17248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:07.299141 17246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:07.299075 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:07.299192 17240 server_base.cc:1034] running on GCE node
I20260925 15:28:07.299666 17240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:07.299937 17240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:07.301126 17240 hybrid_clock.cc:648] HybridClock initialized: now 1790350087301104 us; error 43 us; skew 500 ppm
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:07.303714 17240 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:28:07.304137 17240 webserver.cc:460] Webserver started at http://127.1.163.66:35677/ using document root <none> and password file <none>
I20260925 15:28:07.304354 17240 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:07.304399 17240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:07.304502 17240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:28:07.305613 17240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f390ed59a862b1e687bf85fad5fbdd3f"
server_key_iv: "cf01001a7c41f8fcb8e912456e155445"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.305920 17240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f390ed59a862b1e687bf85fad5fbdd3f"
server_key_iv: "cf01001a7c41f8fcb8e912456e155445"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.307294 17240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:07.308096 17255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.308303 17240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:07.308393 17240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f390ed59a862b1e687bf85fad5fbdd3f"
server_key_iv: "cf01001a7c41f8fcb8e912456e155445"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.308454 17240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:07.323410 17240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:07.325868 17240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:07.325994 17240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:07.326200 17240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:07.326622 17240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:07.326721 17240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.326774 17240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:07.326798 17240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.332924 17240 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:43097
I20260925 15:28:07.333002 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:43097 every 8 connection(s)
I20260925 15:28:07.333333 17240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:07.340736 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17240
I20260925 15:28:07.340871 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:28:07.341125 1677 external_mini_cluster.cc:1428] Setting key d9bac77382489bccad95afd0ffd1f715
I20260925 15:28:07.343322 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.334107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40103 (local address 127.1.163.126:36167)
0925 15:28:07.334168 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.334172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.334396 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:28:07.334925 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.334926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.334939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.334985 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:07.335964 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.336481 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.337506 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.337713 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.337804 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.340147 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:07.340149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:07.340149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:07.340158 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:07.341792 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.342216 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.342217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.342218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.342275 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.342528 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.342529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.342530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.342655 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:07.342738 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.343049 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.343166 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:28:07.343924 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.334090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:40103)
0925 15:28:07.334272 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.334311 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.334824 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.335051 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.335060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.335649 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:07.335854 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.335859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.336598 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.336600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.337403 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.337405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.337831 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.337833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.337951 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.338571 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:07.338580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:07.340051 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:07.341893 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.341901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.341904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.342128 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.342351 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.342352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.342352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.342461 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.342726 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:07.342733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:07.342877 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.343465 (+ 588us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.343661 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 15:28:07.344383 17369 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:07.344485 17369 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:07.344700 17369 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
I20260925 15:28:07.345074 17044 ts_manager.cc:194] Registered new tserver with Master: 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
I20260925 15:28:07.345535 17044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:40103
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:28:07.368726 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:07.452893 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:07.453154 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:07.453194 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:07.454424 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:07.454506 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:07.454555 17374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:28:07.456229 17374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:07.456516 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:07.456801 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:07.462136 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:07.462137 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:07.462234 17374 server_base.cc:1034] running on GCE node
W20260925 15:28:07.462384 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:07.462585 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:07.462893 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:07.464056 17374 hybrid_clock.cc:648] HybridClock initialized: now 1790350087464044 us; error 30 us; skew 500 ppm
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:07.466697 17374 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:28:07.467245 17374 webserver.cc:460] Webserver started at http://127.1.163.67:34863/ using document root <none> and password file <none>
I20260925 15:28:07.467499 17374 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:07.467569 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:07.467720 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:28:07.468863 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "7bd776111e874b26a8b947198e6b3886"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "bfbbae568bed143cb1522cd69b84446a"
server_key_iv: "f4cde867e8ae59aa2680a10e6a76e33c"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.469219 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "7bd776111e874b26a8b947198e6b3886"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "bfbbae568bed143cb1522cd69b84446a"
server_key_iv: "f4cde867e8ae59aa2680a10e6a76e33c"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.470633 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:07.471402 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.471572 17374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:28:07.471666 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "7bd776111e874b26a8b947198e6b3886"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "bfbbae568bed143cb1522cd69b84446a"
server_key_iv: "f4cde867e8ae59aa2680a10e6a76e33c"
server_key_version: "encryptionkey@0"
I20260925 15:28:07.471769 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:07.491742 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:07.494313 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:07.494493 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:07.494755 17374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:07.495163 17374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:07.495223 17374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.495266 17374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:07.495292 17374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:07.501253 17374 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:34309
I20260925 15:28:07.501297 17502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:34309 every 8 connection(s)
I20260925 15:28:07.501654 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:28:07.504033 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17374
I20260925 15:28:07.504156 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:28:07.504339 1677 external_mini_cluster.cc:1428] Setting key 9591847ca1c73e169b7806fcb1ae6e40
Sep 25 15:28:07 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:07.511535 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.502431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:47965 (local address 127.1.163.126:36167)
0925 15:28:07.502496 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.502497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.502713 (+ 216us) server_negotiation.cc:407] Connection header received
0925 15:28:07.503243 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.503244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.503255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.503313 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:07.504375 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.505134 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.506330 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.506542 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.506600 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.508815 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:07.508817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:07.508817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:07.508824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:07.510286 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.510649 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.510650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.510650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.510700 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.510917 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.510918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.510918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.511019 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:07.511070 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.511401 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.511467 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:28:07.511696 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.502412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:47965)
0925 15:28:07.502585 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.502621 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.503137 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.503389 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.503393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.504013 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:07.504263 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.504269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.505244 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.505245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.506245 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.506248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.506620 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.506621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.506724 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.507207 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:07.507216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:07.508727 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:07.510361 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.510366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.510368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.510585 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.510766 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:07.510766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:07.510767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:07.510864 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:07.511075 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:07.511079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:07.511252 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.511356 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.511488 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260925 15:28:07.512195 17503 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:07.512324 17503 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:07.512526 17503 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
I20260925 15:28:07.512974 17044 ts_manager.cc:194] Registered new tserver with Master: 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309)
I20260925 15:28:07.513350 17044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:47965
I20260925 15:28:07.516211 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:28:07.523433 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.517580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.1.163.126:36167)
0925 15:28:07.517652 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.517654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.517726 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:28:07.517762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.517763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.517772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.517820 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:07.518362 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.518839 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.519541 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.519717 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.519765 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.521035 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:07.521036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:07.521036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:07.521042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:07.522330 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.522670 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.522671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.522671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.522719 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:07.522935 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:07.522936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:07.522936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:07.523076 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:07.523152 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.523278 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.523339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:28:07.525347 17044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:28:07.527155 17044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:28:07.534687 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.1.163.65:39311)
0925 15:28:07.530921 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.530925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.530937 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:28:07.530972 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.530974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.530990 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.531315 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:07.531390 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532471 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.533539 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.534197 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.534277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.534337 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.534533 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.534577 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":70,"threads_started":1}
I20260925 15:28:07.534683 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:39311 (local address 127.0.0.1:50292)
0925 15:28:07.530450 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.530460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.530521 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.531083 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.531084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.531086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:07.531146 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.531149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532651 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.532652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.533437 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.533440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.534307 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.534308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.534403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.534419 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.534505 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.534565 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260925 15:28:07.534966 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34309 (local address 127.0.0.1:48502)
0925 15:28:07.530627 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.530636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.530672 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.531038 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.531042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.531045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:07.531211 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.531212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532576 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.532578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.533768 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.533771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.534712 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.534713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.534796 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.534801 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.534835 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.534877 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":101,"thread_start_us":90,"threads_started":1}
I20260925 15:28:07.535368 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.1.163.67:34309)
0925 15:28:07.530645 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.530651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.530665 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:28:07.530884 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.530887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.530905 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.531125 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:07.531276 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532427 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.533863 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.534597 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.535138 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.535221 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.535253 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.535296 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260925 15:28:07.535439 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:43097 (local address 127.0.0.1:48556)
0925 15:28:07.530392 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.530439 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.530521 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.531129 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.531130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.531132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:07.531198 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.531201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532230 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.532232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.533877 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.533879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.535169 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.535170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.535249 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.535254 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.535310 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.535352 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260925 15:28:07.535512 17169 tablet_service.cc:1467] Processing CreateTablet for tablet 79b19099a899490b8ee6520abdcd9b11 (DEFAULT_TABLE table=test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:07.535699 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.530695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.1.163.66:43097)
0925 15:28:07.530950 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.530955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.530967 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:28:07.531024 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.531027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.531045 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.531133 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:07.531318 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.532087 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.533950 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.534602 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.534673 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.534740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.535559 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.535621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260925 15:28:07.535907 17169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b19099a899490b8ee6520abdcd9b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:07.537237 17303 tablet_service.cc:1467] Processing CreateTablet for tablet 79b19099a899490b8ee6520abdcd9b11 (DEFAULT_TABLE table=test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:07.537352 17437 tablet_service.cc:1467] Processing CreateTablet for tablet 79b19099a899490b8ee6520abdcd9b11 (DEFAULT_TABLE table=test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:07.537580 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b19099a899490b8ee6520abdcd9b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:07.537689 17437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b19099a899490b8ee6520abdcd9b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:07.539263 17520 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap starting.
I20260925 15:28:07.540084 17520 tablet_bootstrap.cc:654] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:07.540621 17521 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Bootstrap starting.
I20260925 15:28:07.540930 17520 log.cc:824] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:07.540990 17522 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Bootstrap starting.
I20260925 15:28:07.541276 17521 tablet_bootstrap.cc:654] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:07.541695 17520 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: No bootstrap required, opened a new log
I20260925 15:28:07.541771 17522 tablet_bootstrap.cc:654] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:07.541779 17520 ts_tablet_manager.cc:1397] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:28:07.541837 17521 log.cc:824] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:07.542279 17522 log.cc:824] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:07.542641 17521 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: No bootstrap required, opened a new log
I20260925 15:28:07.542723 17521 ts_tablet_manager.cc:1397] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:28:07.543066 17522 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: No bootstrap required, opened a new log
I20260925 15:28:07.543134 17522 ts_tablet_manager.cc:1397] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:28:07.543612 17520 raft_consensus.cc:348] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.543783 17520 raft_consensus.cc:374] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:07.543828 17520 raft_consensus.cc:729] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Initialized, Role: FOLLOWER
I20260925 15:28:07.543941 17520 consensus_queue.cc:260] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.544227 17520 ts_tablet_manager.cc:1428] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:28:07.544502 17235 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
I20260925 15:28:07.545140 17522 raft_consensus.cc:348] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.545320 17522 raft_consensus.cc:374] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:07.545356 17522 raft_consensus.cc:729] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d2a2aa986a44d288aa35bac2e79f2c, State: Initialized, Role: FOLLOWER
I20260925 15:28:07.545261 17521 raft_consensus.cc:348] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.545423 17521 raft_consensus.cc:374] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:07.545459 17521 raft_consensus.cc:729] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd776111e874b26a8b947198e6b3886, State: Initialized, Role: FOLLOWER
I20260925 15:28:07.545493 17522 consensus_queue.cc:260] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.545701 17521 consensus_queue.cc:260] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.545783 17522 ts_tablet_manager.cc:1428] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:28:07.545802 17369 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
I20260925 15:28:07.545989 17521 ts_tablet_manager.cc:1428] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:28:07.546013 17503 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
W20260925 15:28:07.584036 17370 tablet.cc:2369] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:28:07.673264 17236 tablet.cc:2369] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:28:07.752310 17504 tablet.cc:2369] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:28:07.763537 17526 raft_consensus.cc:482] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:07.763674 17526 raft_consensus.cc:504] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.764218 17526 leader_election.cc:288] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097), 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309)
I20260925 15:28:07.768707 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.764613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:55169 (local address 127.1.163.66:43097)
0925 15:28:07.764672 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.764675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.764688 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:28:07.764745 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.764746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.764755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.764833 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:07.765070 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.765845 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.767556 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.768229 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.768304 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.768315 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.768585 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.768636 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:28:07.768707 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.764632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:46235 (local address 127.1.163.67:34309)
0925 15:28:07.764676 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.764678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.764828 (+ 150us) server_negotiation.cc:407] Connection header received
0925 15:28:07.764911 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.764912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.764921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.764960 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:07.765149 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.765847 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.767555 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.768185 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.768304 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.768315 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.768577 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.768630 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:28:07.768740 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.764523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34309 (local address 127.1.163.65:46235)
0925 15:28:07.764697 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.764707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.764837 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.764996 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.764997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.764999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:07.765071 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.765073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.765964 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.765965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.767411 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.767415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.768353 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.768354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.768450 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.768458 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.768538 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.768618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":115,"thread_start_us":108,"threads_started":1}
I20260925 15:28:07.768745 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.764422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:43097 (local address 127.1.163.65:55169)
0925 15:28:07.764596 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.764610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:07.764674 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:07.764864 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:07.764865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:07.764867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:07.764982 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.764984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.765964 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.765965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.767411 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:07.767415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.768351 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.768353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:07.768450 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.768460 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:28:07.768576 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:28:07.768618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":99,"thread_start_us":87,"threads_started":1}
I20260925 15:28:07.769320 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b19099a899490b8ee6520abdcd9b11" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd776111e874b26a8b947198e6b3886" is_pre_election: true
I20260925 15:28:07.769342 17323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b19099a899490b8ee6520abdcd9b11" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" is_pre_election: true
I20260925 15:28:07.769496 17457 raft_consensus.cc:2454] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 0.
I20260925 15:28:07.769496 17323 raft_consensus.cc:2454] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 0.
I20260925 15:28:07.769747 17122 leader_election.cc:302] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa, 7bd776111e874b26a8b947198e6b3886; no voters:
I20260925 15:28:07.769948 17526 raft_consensus.cc:2781] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:28:07.770020 17526 raft_consensus.cc:482] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:28:07.770040 17526 raft_consensus.cc:3037] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:07.770803 17526 raft_consensus.cc:504] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.770970 17526 leader_election.cc:288] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 election: Requested vote from peers 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097), 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309)
I20260925 15:28:07.771181 17323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b19099a899490b8ee6520abdcd9b11" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
I20260925 15:28:07.771183 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b19099a899490b8ee6520abdcd9b11" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd776111e874b26a8b947198e6b3886"
I20260925 15:28:07.771274 17457 raft_consensus.cc:3037] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:07.771274 17323 raft_consensus.cc:3037] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:07.772208 17457 raft_consensus.cc:2454] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 1.
I20260925 15:28:07.772208 17323 raft_consensus.cc:2454] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 1.
I20260925 15:28:07.772399 17122 leader_election.cc:302] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa, 7bd776111e874b26a8b947198e6b3886; no voters:
I20260925 15:28:07.772516 17526 raft_consensus.cc:2781] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:28:07.772739 17526 raft_consensus.cc:686] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 LEADER]: Becoming Leader. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Running, Role: LEADER
I20260925 15:28:07.772845 17526 consensus_queue.cc:237] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:07.773811 17044 catalog_manager.cc:5495] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65). New cstate: current_term: 1 leader_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } health_report { overall_health: UNKNOWN } } }
I20260925 15:28:07.822185 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.819366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.1.163.65:39311)
0925 15:28:07.819435 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:07.819437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:07.819446 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:28:07.819518 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:07.819519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:07.819539 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:07.819593 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:28:07.819904 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.820421 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.821309 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:07.821515 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:07.821593 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:07.821766 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:28:07.821872 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:28:07.821944 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:07.822036 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:28:07.822057 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:28:07.824254 17323 raft_consensus.cc:1261] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Refusing update from remote peer 193bc2deeaf342babe54a2a3edcbe7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:28:07.824263 17457 raft_consensus.cc:1261] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Refusing update from remote peer 193bc2deeaf342babe54a2a3edcbe7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:28:07.824671 17526 consensus_queue.cc:1035] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:28:07.824832 17531 consensus_queue.cc:1035] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:28:07.828341 17535 mvcc.cc:205] Tried to move back new op lower bound from 7333273959725215744 to 7333273959518408704. Current Snapshot: MvccSnapshot[applied={T|T < 7333273959725215744}]
I20260925 15:28:07.830065 17043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:28:07.830169 17043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:28:07.831013 17043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Sending DeleteTablet for 3 replicas of tablet 79b19099a899490b8ee6520abdcd9b11
I20260925 15:28:07.831372 17303 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b19099a899490b8ee6520abdcd9b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:07 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:48556
I20260925 15:28:07.831363 17169 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b19099a899490b8ee6520abdcd9b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:07 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:50292
I20260925 15:28:07.831684 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17106
I20260925 15:28:07.831682 17437 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b19099a899490b8ee6520abdcd9b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:07 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:48502
I20260925 15:28:07.832638 17545 tablet_replica.cc:330] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: stopping tablet replica
I20260925 15:28:07.832762 17545 raft_consensus.cc:2229] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:28:07.832873 17545 raft_consensus.cc:2258] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:07.833364 17545 ts_tablet_manager.cc:1905] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:07.833642 17546 tablet_replica.cc:330] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: stopping tablet replica
I20260925 15:28:07.833767 17546 raft_consensus.cc:2229] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:28:07.833912 17546 raft_consensus.cc:2258] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:07.834316 17546 ts_tablet_manager.cc:1905] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:07.835326 17545 ts_tablet_manager.cc:1918] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:28:07.835404 17545 log.cc:1196] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/79b19099a899490b8ee6520abdcd9b11
I20260925 15:28:07.835671 17545 ts_tablet_manager.cc:1939] T 79b19099a899490b8ee6520abdcd9b11 P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting consensus metadata
I20260925 15:28:07.836174 17546 ts_tablet_manager.cc:1918] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:28:07.836155 17029 catalog_manager.cc:4841] TS 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): tablet 79b19099a899490b8ee6520abdcd9b11 (table test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]) successfully deleted
I20260925 15:28:07.836246 17546 log.cc:1196] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/79b19099a899490b8ee6520abdcd9b11
I20260925 15:28:07.836591 17546 ts_tablet_manager.cc:1939] T 79b19099a899490b8ee6520abdcd9b11 P 7bd776111e874b26a8b947198e6b3886: Deleting consensus metadata
I20260925 15:28:07.837045 17028 catalog_manager.cc:4841] TS 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): tablet 79b19099a899490b8ee6520abdcd9b11 (table test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]) successfully deleted
W20260925 15:28:07.838524 17030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:39311 (error 108)
I20260925 15:28:07.839109 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17240
I20260925 15:28:07.839140 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:07.838995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:39311 (local address 127.0.0.1:50306)
0925 15:28:07.839054 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:07.839098 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:28:07.839274 17030 catalog_manager.cc:4584] TS 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65:39311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79b19099a899490b8ee6520abdcd9b11: Network error: Client connection negotiation failed: client connection to 127.1.163.65:39311: connect: Connection refused (error 111)
I20260925 15:28:07.843968 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17374
I20260925 15:28:07.848753 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17012
I20260925 15:28:27.854552 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--webserver_interface=127.1.163.126
--webserver_port=44675
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:27.939100 17548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:27.939329 17548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:27.939353 17548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:27.940973 17548 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:28:27.941033 17548 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:28:27.941056 17548 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:28:27.941087 17548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:28:27.941107 17548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:27.941136 17548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:27.941157 17548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:28:27.941187 17548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:28:27.942780 17548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:27.943058 17548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:27.943333 17548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:27.952358 17557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:27.952373 17555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:27.952517 17548 server_base.cc:1034] running on GCE node
W20260925 15:28:27.952389 17554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:27.952922 17548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:27.953230 17548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:27.954406 17548 hybrid_clock.cc:648] HybridClock initialized: now 1790350107954386 us; error 39 us; skew 500 ppm
Sep 25 15:28:27 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:27.961741 17548 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:28:27.962283 17548 webserver.cc:460] Webserver started at http://127.1.163.126:44675/ using document root <none> and password file <none>
I20260925 15:28:27.962558 17548 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:27.962618 17548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:27.964193 17548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:27.965040 17564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:27.965258 17548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:28:27.965389 17548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "1ba9d19c66c74c9496a7b3da228f5406"
format_stamp: "Formatted at 2026-09-25 15:28:06 on dist-test-slave-hvv5"
server_key: "146f9f0ea032a2f5277793edd14b5c8b"
server_key_iv: "ec0080601d7550e88e5ea8714db095dd"
server_key_version: "encryptionkey@0"
I20260925 15:28:27.965853 17548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:27.982525 17548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:27.984958 17548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:27.985119 17548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:27.989315 17548 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36167
I20260925 15:28:27.989362 17616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36167 every 8 connection(s)
I20260925 15:28:27.989779 17548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:28:27.990339 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17548
I20260925 15:28:27.990643 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:39311
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=34083
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:28:27.991482 17617 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:28:27.992228 17617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap starting.
I20260925 15:28:28.000919 17617 log.cc:824] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:28.003016 17617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:28:28.003381 17617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap complete.
I20260925 15:28:28.005818 17617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:28.006083 17617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Initialized, Role: FOLLOWER
I20260925 15:28:28.006295 17617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:28.006397 17617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:28:28.006474 17617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:28:28.006525 17617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:28:28.007611 17617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:28.007740 17617 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba9d19c66c74c9496a7b3da228f5406; no voters:
I20260925 15:28:28.007963 17617 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:28:28.008001 17621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:28:28.008113 17621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 LEADER]: Becoming Leader. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Running, Role: LEADER
I20260925 15:28:28.008203 17621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:28.008255 17617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:28:28.008489 17622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:28.008602 17622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:28.008857 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba9d19c66c74c9496a7b3da228f5406. Latest consensus state: current_term: 2 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:28.008956 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:28.009088 17625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:28:28.009902 17625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:28:28.010010 17625 catalog_manager.cc:1259] Loaded cluster ID: b7ee3d5b55dc4441b7f56e641299c522
I20260925 15:28:28.010049 17625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:28:28.011353 17625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:28:28.011545 17625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Loaded TSK: 0
I20260925 15:28:28.024608 17625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Generated new TSK 1
I20260925 15:28:28.024734 17625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:28:28.082597 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:28.082840 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:28.082870 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:28.084234 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:28.084291 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:28.084322 17619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:28:28.086405 17619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:39311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=34083
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:28.086732 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:28.087049 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:28.092568 17645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.092561 17644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.092566 17647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:28.093010 17619 server_base.cc:1034] running on GCE node
I20260925 15:28:28.093235 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:28.093523 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:28.094729 17619 hybrid_clock.cc:648] HybridClock initialized: now 1790350108094703 us; error 41 us; skew 500 ppm
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:28.097905 17619 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:28:28.098556 17619 webserver.cc:460] Webserver started at http://127.1.163.65:34083/ using document root <none> and password file <none>
I20260925 15:28:28.098889 17619 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:28.098991 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:28.100661 17619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:28.101657 17654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.101926 17619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:28.102051 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "193bc2deeaf342babe54a2a3edcbe7fa"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f1a430c04de8f5a59d5eca0baaba9426"
server_key_iv: "0cd6f75f72595f41e62f9ecf95c1af3f"
server_key_version: "encryptionkey@0"
I20260925 15:28:28.102444 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:28.112813 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:28.115334 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:28.115513 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:28.115808 17619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:28.116384 17661 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:28:28.118094 17619 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:28:28.118149 17619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:28:28.118196 17619 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:28:28.119129 17619 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:28:28.119182 17619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:28:28.119223 17661 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap starting.
I20260925 15:28:28.127279 17619 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:39311
I20260925 15:28:28.127825 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:28:28.130825 17661 log.cc:824] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:28.131178 17768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:39311 every 8 connection(s)
I20260925 15:28:28.133590 17661 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:28:28.133920 17661 tablet_bootstrap.cc:492] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap complete.
I20260925 15:28:28.134967 17661 ts_tablet_manager.cc:1397] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260925 15:28:28.136221 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17619
I20260925 15:28:28.136159 17661 raft_consensus.cc:348] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:28.136492 17661 raft_consensus.cc:729] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Initialized, Role: FOLLOWER
I20260925 15:28:28.136582 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:43097
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=35677
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:28:28.136662 17661 consensus_queue.cc:260] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } } peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } }
I20260925 15:28:28.136986 17661 ts_tablet_manager.cc:1428] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:28.144191 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.128647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:50477)
0925 15:28:28.128922 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.128965 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.129604 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.131986 (+ 2382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.131991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.132377 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:28.132695 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.132703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.134477 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.134479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.135773 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.135777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.137258 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.137259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.137362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.137992 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:28.138001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:28.139510 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:28.142149 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.142157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.142160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.142504 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.142832 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.142833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.142833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.142974 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.143489 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:28.143496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:28.143658 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.143782 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.143968 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260925 15:28:28.145130 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.128759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:50477 (local address 127.1.163.126:36167)
0925 15:28:28.131502 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.131512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.131555 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:28:28.131695 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.131702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.131762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.131923 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:28.132842 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.134313 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.135885 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.137018 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.137177 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.139633 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:28.139638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:28.139638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:28.139651 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:28.142030 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.142606 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.142610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.142611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.142750 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.143053 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.143054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.143054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.143405 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:28.143505 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.143881 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.144046 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":101,"threads_started":1}
I20260925 15:28:28.145856 17769 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:28.145974 17769 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:28.146202 17769 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
I20260925 15:28:28.147265 17580 ts_manager.cc:194] Registered new tserver with Master: 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65:39311)
I20260925 15:28:28.148855 17580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:50477
I20260925 15:28:28.150022 17769 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
I20260925 15:28:28.158336 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.148073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:39311 (local address 127.0.0.1:51656)
0925 15:28:28.149289 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.149329 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.149406 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.152970 (+ 3564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.152973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.152975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:28.153087 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.153109 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.154890 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.154891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.156473 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.156477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.157843 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.157844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.157956 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.158007 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.158097 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.158186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":102,"threads_started":1}
I20260925 15:28:28.158941 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.148380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.1.163.65:39311)
0925 15:28:28.152677 (+ 4297us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.152683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.152695 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:28:28.152759 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.152761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.152778 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.152914 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:28.153241 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.154728 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.156703 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.157673 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.157795 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.158755 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.158789 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.158843 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4232,"thread_start_us":139,"threads_started":1}
I20260925 15:28:28.159307 17685 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b19099a899490b8ee6520abdcd9b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:07 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51656
I20260925 15:28:28.160096 17778 tablet_replica.cc:330] stopping tablet replica
I20260925 15:28:28.160233 17778 raft_consensus.cc:2229] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:28:28.160303 17778 raft_consensus.cc:2258] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:28.160780 17778 ts_tablet_manager.cc:1905] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:28.163522 17778 ts_tablet_manager.cc:1918] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:28:28.163586 17778 log.cc:1196] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal/wals/79b19099a899490b8ee6520abdcd9b11
I20260925 15:28:28.163858 17778 ts_tablet_manager.cc:1939] T 79b19099a899490b8ee6520abdcd9b11 P 193bc2deeaf342babe54a2a3edcbe7fa: Deleting consensus metadata
I20260925 15:28:28.164440 17567 catalog_manager.cc:4841] TS 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65:39311): tablet 79b19099a899490b8ee6520abdcd9b11 (table test-table [id=297330d16fbd41dd9dd4f9e78552f9f2]) successfully deleted
W20260925 15:28:28.231104 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:28.231392 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:28.231438 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:28.232753 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:28.232832 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:28.232879 17775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:28:28.234449 17775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:43097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=35677
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:28.234746 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:28.234999 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:28.240373 17783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.240356 17786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.240350 17784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:28.241005 17775 server_base.cc:1034] running on GCE node
I20260925 15:28:28.241201 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:28.241434 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:28.242589 17775 hybrid_clock.cc:648] HybridClock initialized: now 1790350108242575 us; error 28 us; skew 500 ppm
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:28.245347 17775 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:28:28.245870 17775 webserver.cc:460] Webserver started at http://127.1.163.66:35677/ using document root <none> and password file <none>
I20260925 15:28:28.246104 17775 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:28.246207 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:28.247611 17775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:28.248358 17793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.248585 17775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:28.248696 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f390ed59a862b1e687bf85fad5fbdd3f"
server_key_iv: "cf01001a7c41f8fcb8e912456e155445"
server_key_version: "encryptionkey@0"
I20260925 15:28:28.249063 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:28.275159 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:28.277741 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:28.277925 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:28.278185 17775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:28.278512 17775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:28.278568 17775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.278602 17775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:28.278650 17775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.284852 17775 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:43097
I20260925 15:28:28.284914 17906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:43097 every 8 connection(s)
I20260925 15:28:28.285303 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:28.294075 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17775
I20260925 15:28:28.294459 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:34309
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=34863
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:28:28.296253 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.286169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:55035 (local address 127.1.163.126:36167)
0925 15:28:28.286244 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.286247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.286520 (+ 273us) server_negotiation.cc:407] Connection header received
0925 15:28:28.287041 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.287042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.287059 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.287120 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:28.288149 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.288827 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.289912 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.290130 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.290195 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.292707 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:28.292708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:28.292708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:28.292728 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:28.294170 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.294871 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.294872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.294873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.294948 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.295322 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.295324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.295324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.295549 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:28.295611 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.296026 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.296147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:28:28.296429 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.286155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:55035)
0925 15:28:28.286373 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.286409 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.286916 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.287201 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.287204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.287839 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:28.288037 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.288043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.288916 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.288917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.289809 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.289811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.290228 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.290229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.290340 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.290951 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:28.290960 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:28.292521 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:28.294286 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.294328 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.294331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.294750 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.295037 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.295038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.295038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.295192 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.295630 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:28.295638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:28.295840 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.295986 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.296182 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260925 15:28:28.297093 17907 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:28.297219 17907 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:28.297431 17907 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
I20260925 15:28:28.297887 17580 ts_manager.cc:194] Registered new tserver with Master: 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
I20260925 15:28:28.298316 17580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:55035
W20260925 15:28:28.380537 17910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:28.380893 17910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:28.380937 17910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:28.382416 17910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:28.382490 17910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:28.382539 17910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:28:28.384208 17910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:34309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=34863
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:28.384497 17910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:28.384776 17910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:28.390182 17918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.390189 17915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:28.390434 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:28.390445 17910 server_base.cc:1034] running on GCE node
I20260925 15:28:28.390698 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:28.391000 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:28.392184 17910 hybrid_clock.cc:648] HybridClock initialized: now 1790350108392158 us; error 40 us; skew 500 ppm
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:28.394906 17910 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:28:28.395396 17910 webserver.cc:460] Webserver started at http://127.1.163.67:34863/ using document root <none> and password file <none>
I20260925 15:28:28.395664 17910 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:28.395748 17910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:28.397321 17910 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:28:28.398018 17925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.398296 17910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:28:28.398407 17910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "7bd776111e874b26a8b947198e6b3886"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "bfbbae568bed143cb1522cd69b84446a"
server_key_iv: "f4cde867e8ae59aa2680a10e6a76e33c"
server_key_version: "encryptionkey@0"
I20260925 15:28:28.398810 17910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:28.414577 17910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:28.417209 17910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:28.417412 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:28.417762 17910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:28.418133 17910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:28.418192 17910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.418244 17910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:28.418274 17910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.424793 17910 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:34309
I20260925 15:28:28.424870 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:34309 every 8 connection(s)
I20260925 15:28:28.425269 17910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:28:28.430764 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 17910
Sep 25 15:28:28 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350108, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:28:28.435139 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.426191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53249 (local address 127.1.163.126:36167)
0925 15:28:28.426277 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.426280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.426478 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:28:28.427001 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.427002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.427012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.427076 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:28.428042 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.428522 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.429633 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.429817 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.429882 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.432210 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:28.432212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:28.432212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:28.432220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:28.433723 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.434150 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.434152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.434152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.434214 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.434480 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.434481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.434482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.434593 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:28.434673 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.434955 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.435037 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:28:28.435360 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.426146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:53249)
0925 15:28:28.426330 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.426376 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.426913 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.427132 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.427135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.427739 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:28.427947 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.427952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.428667 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.428668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.429542 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.429545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.429922 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.429923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.430032 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.430575 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:28.430585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:28.432109 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:28.433827 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.433835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.433837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.434071 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.434291 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:28.434292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:28.434292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:28.434411 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:28.434665 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:28.434671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:28.434817 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.434934 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.435106 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260925 15:28:28.435935 18039 heartbeater.cc:347] Connected to a master server at 127.1.163.126:36167
I20260925 15:28:28.436048 18039 heartbeater.cc:464] Registering TS with master...
I20260925 15:28:28.436233 18039 heartbeater.cc:511] Master 127.1.163.126:36167 requested a full tablet report, sending...
I20260925 15:28:28.436859 17580 ts_manager.cc:194] Registered new tserver with Master: 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309)
I20260925 15:28:28.437326 17580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:53249
I20260925 15:28:28.438350 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.431279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.1.163.126:36167)
0925 15:28:28.431399 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.431401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.431474 (+ 73us) server_negotiation.cc:407] Connection header received
0925 15:28:28.431568 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.431569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.431583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.431650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:28.432281 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.432836 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.433608 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.433804 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.433877 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.435055 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:28.435056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:28.435057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:28.435064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:28.437057 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.437518 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.437519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.437519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.437571 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.437830 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.437831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.437831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.437939 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:28.438133 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.438171 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.438248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260925 15:28:28.439018 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:28:28.441627 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.439328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.1.163.126:36167)
0925 15:28:28.439364 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.439366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.439509 (+ 143us) server_negotiation.cc:407] Connection header received
0925 15:28:28.439552 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.439553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.439581 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.439637 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:28:28.439762 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.440209 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.441001 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.441183 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.441239 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.441409 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:28:28.441449 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:28:28.441563 (+ 114us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60130: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:28:28.441679 18043 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60130: token expired
W20260925 15:28:28.441783 17508 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.163.126:36167)
I20260925 15:28:28.448068 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.442018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.1.163.126:36167)
0925 15:28:28.442054 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.442056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.442066 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:28:28.442125 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.442125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.442132 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.442168 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:28.442601 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.443061 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.443776 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.443978 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.444030 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.445276 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:28.445278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:28.445278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:28.445285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:28.446754 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.447071 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.447072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.447072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.447123 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:28.447365 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:28.447367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:28.447367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:28.447531 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:28.447613 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.447868 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.447936 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:28:28.449517 17580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:28:28.451400 17580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:28:28.453895 17685 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b (DEFAULT_TABLE table=test-table [id=177abfe8e23f46d1b8c284b9a933d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:28.454114 17685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6f51d1ff24417ba63f8c2763b49f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:28.455663 17661 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap starting.
I20260925 15:28:28.456097 17661 tablet_bootstrap.cc:654] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:28.456748 17661 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: No bootstrap required, opened a new log
I20260925 15:28:28.456800 17661 ts_tablet_manager.cc:1397] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:28.456918 17661 raft_consensus.cc:348] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.457006 17661 raft_consensus.cc:374] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:28.457067 17661 raft_consensus.cc:729] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Initialized, Role: FOLLOWER
I20260925 15:28:28.457111 17661 consensus_queue.cc:260] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.457209 17661 ts_tablet_manager.cc:1428] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:28.457818 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.453569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34309 (local address 127.0.0.1:42668)
0925 15:28:28.453626 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.453638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.453678 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.454319 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.454320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.454322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:28.454370 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.454372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.455378 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.455380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.456516 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.456519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.457555 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.457556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.457623 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.457627 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.457684 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.457739 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":3}
I20260925 15:28:28.457831 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.453960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.1.163.67:34309)
0925 15:28:28.454126 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.454130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.454140 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:28:28.454184 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.454202 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.454218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.454301 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:28.454485 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.455290 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.456683 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.457451 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.457555 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.457622 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.457697 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.457741 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260925 15:28:28.458550 17973 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b (DEFAULT_TABLE table=test-table [id=177abfe8e23f46d1b8c284b9a933d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:28.458884 17973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6f51d1ff24417ba63f8c2763b49f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:28.459232 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.453711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.1.163.66:43097)
0925 15:28:28.455712 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.455717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.455728 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:28:28.455769 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.455771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.455787 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.455876 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:28.455992 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.457083 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.458347 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.458885 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.458986 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.459043 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.459141 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.459174 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1962,"thread_start_us":141,"threads_started":1}
I20260925 15:28:28.459247 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.453569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:43097 (local address 127.0.0.1:58788)
0925 15:28:28.453843 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.453853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.453905 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.455875 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.455876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.455878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:28.455925 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.455927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.457181 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.457182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.458254 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.458256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.458994 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.458995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.459061 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.459065 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.459127 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.459169 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":91,"thread_start_us":132,"threads_started":1}
I20260925 15:28:28.459964 17841 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b (DEFAULT_TABLE table=test-table [id=177abfe8e23f46d1b8c284b9a933d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:28:28.460314 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6f51d1ff24417ba63f8c2763b49f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:28:28.462175 18049 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Bootstrap starting.
I20260925 15:28:28.462869 18049 tablet_bootstrap.cc:654] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:28.463501 18050 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Bootstrap starting.
I20260925 15:28:28.463510 18049 log.cc:824] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:28.464154 18050 tablet_bootstrap.cc:654] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Neither blocks nor log segments found. Creating new log.
I20260925 15:28:28.464262 18049 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: No bootstrap required, opened a new log
I20260925 15:28:28.464365 18049 ts_tablet_manager.cc:1397] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:28:28.464772 18050 log.cc:824] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:28.465457 18050 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: No bootstrap required, opened a new log
I20260925 15:28:28.465512 18050 ts_tablet_manager.cc:1397] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:28:28.466009 18049 raft_consensus.cc:348] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.466178 18049 raft_consensus.cc:374] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:28.466219 18049 raft_consensus.cc:729] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd776111e874b26a8b947198e6b3886, State: Initialized, Role: FOLLOWER
I20260925 15:28:28.466391 18049 consensus_queue.cc:260] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.466734 18049 ts_tablet_manager.cc:1428] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:28:28.466734 18039 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
I20260925 15:28:28.467485 18050 raft_consensus.cc:348] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.467676 18050 raft_consensus.cc:374] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:28:28.467723 18050 raft_consensus.cc:729] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d2a2aa986a44d288aa35bac2e79f2c, State: Initialized, Role: FOLLOWER
I20260925 15:28:28.467850 18050 consensus_queue.cc:260] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.468115 17907 heartbeater.cc:503] Master 127.1.163.126:36167 was elected leader, sending a full tablet report...
I20260925 15:28:28.468127 18050 ts_tablet_manager.cc:1428] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:28:28.489193 17774 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:28.489408 17774 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.489799 17774 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
I20260925 15:28:28.493214 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.490148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:50375 (local address 127.1.163.66:43097)
0925 15:28:28.490206 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.490208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.490352 (+ 144us) server_negotiation.cc:407] Connection header received
0925 15:28:28.490427 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.490428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.490439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.490517 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:28.490668 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.491242 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492271 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.492819 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492881 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.492889 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.493086 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.493136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:28:28.493255 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.490042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34309 (local address 127.1.163.65:42185)
0925 15:28:28.490121 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.490134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.490205 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.490367 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.490368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.490370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:28.490700 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.490702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.491580 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.491581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.492345 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.492348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.492977 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.493034 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.493058 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.493119 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.493164 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:28:28.493274 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.490058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:43097 (local address 127.1.163.65:50375)
0925 15:28:28.490277 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.490287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:28.490368 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:28.490523 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:28.490524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:28.490526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:28:28.490591 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.490593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.491359 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.491360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.492172 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:28.492175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.492919 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:28.493003 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.493007 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:28:28.493060 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:28:28.493126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":131,"threads_started":1}
I20260925 15:28:28.493255 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.490129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:42185 (local address 127.1.163.67:34309)
0925 15:28:28.490209 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.490211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.490221 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:28:28.490279 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.490280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.490289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.490365 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:28:28.490786 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.491501 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492433 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.492897 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.492973 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.492981 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.493136 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.493177 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:28:28.493657 17993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6f51d1ff24417ba63f8c2763b49f0b" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd776111e874b26a8b947198e6b3886" is_pre_election: true
I20260925 15:28:28.493690 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6f51d1ff24417ba63f8c2763b49f0b" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" is_pre_election: true
I20260925 15:28:28.493840 17861 raft_consensus.cc:2454] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 0.
I20260925 15:28:28.493842 17993 raft_consensus.cc:2454] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 0.
I20260925 15:28:28.494078 17656 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa, 48d2a2aa986a44d288aa35bac2e79f2c; no voters:
I20260925 15:28:28.494240 17774 raft_consensus.cc:2781] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:28:28.494280 17774 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:28:28.494297 17774 raft_consensus.cc:3037] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:28.494982 17774 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.495136 17774 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 election: Requested vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
I20260925 15:28:28.495319 17993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6f51d1ff24417ba63f8c2763b49f0b" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd776111e874b26a8b947198e6b3886"
I20260925 15:28:28.495342 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6f51d1ff24417ba63f8c2763b49f0b" candidate_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
I20260925 15:28:28.495428 17993 raft_consensus.cc:3037] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:28.495424 17861 raft_consensus.cc:3037] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:28:28.496367 17861 raft_consensus.cc:2454] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 1.
I20260925 15:28:28.496522 17993 raft_consensus.cc:2454] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193bc2deeaf342babe54a2a3edcbe7fa in term 1.
I20260925 15:28:28.496580 17656 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa, 48d2a2aa986a44d288aa35bac2e79f2c; no voters:
I20260925 15:28:28.496662 17774 raft_consensus.cc:2781] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:28:28.496819 17774 raft_consensus.cc:686] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 LEADER]: Becoming Leader. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Running, Role: LEADER
I20260925 15:28:28.496908 17774 consensus_queue.cc:237] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:28.497921 17580 catalog_manager.cc:5495] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65). New cstate: current_term: 1 leader_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } health_report { overall_health: HEALTHY } } }
I20260925 15:28:28.505319 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.502870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.1.163.65:39311)
0925 15:28:28.502913 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:28.502915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:28.502925 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:28:28.502988 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:28.502988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:28.503005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:28.503071 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:28:28.503166 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.503618 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.504476 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:28.504771 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:28.504864 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:28.505010 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:28:28.505124 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:28:28.505180 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:28.505226 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:28:28.505251 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:28:28.507051 17993 raft_consensus.cc:1261] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Refusing update from remote peer 193bc2deeaf342babe54a2a3edcbe7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:28:28.507052 17861 raft_consensus.cc:1261] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Refusing update from remote peer 193bc2deeaf342babe54a2a3edcbe7fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:28:28.507444 18056 consensus_queue.cc:1035] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:28:28.507576 17774 consensus_queue.cc:1035] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:28:28.510326 18058 mvcc.cc:205] Tried to move back new op lower bound from 7333274044442124288 to 7333274044403965952. Current Snapshot: MvccSnapshot[applied={T|T < 7333274044442124288}]
I20260925 15:28:28.513397 17580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:28:28.513491 17580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:28:28.514353 17580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Sending DeleteTablet for 3 replicas of tablet 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:28.514683 17973 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:28 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42668
I20260925 15:28:28.514761 17685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:28 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:51656
I20260925 15:28:28.514744 17841 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6f51d1ff24417ba63f8c2763b49f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:28:28 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:58788
I20260925 15:28:28.514847 17778 tablet_replica.cc:330] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: stopping tablet replica
I20260925 15:28:28.514901 17778 raft_consensus.cc:2229] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:28:28.515010 17778 raft_consensus.cc:2258] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:28.515033 18067 tablet_replica.cc:330] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: stopping tablet replica
I20260925 15:28:28.515134 18067 raft_consensus.cc:2229] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:28:28.515152 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17619
I20260925 15:28:28.515246 18067 raft_consensus.cc:2258] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:28.515354 17778 ts_tablet_manager.cc:1905] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:28.517505 18068 tablet_replica.cc:330] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: stopping tablet replica
I20260925 15:28:28.517635 18068 raft_consensus.cc:2229] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:28:28.517748 18068 raft_consensus.cc:2258] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:28:28.518255 18067 ts_tablet_manager.cc:1905] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:28.518267 18068 ts_tablet_manager.cc:1905] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:28:28.520013 18068 ts_tablet_manager.cc:1918] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:28:28.520119 18068 log.cc:1196] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:28.520220 18067 ts_tablet_manager.cc:1918] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:28:28.520298 18067 log.cc:1196] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:28.520505 18068 ts_tablet_manager.cc:1939] T 8e6f51d1ff24417ba63f8c2763b49f0b P 48d2a2aa986a44d288aa35bac2e79f2c: Deleting consensus metadata
I20260925 15:28:28.520627 18067 ts_tablet_manager.cc:1939] T 8e6f51d1ff24417ba63f8c2763b49f0b P 7bd776111e874b26a8b947198e6b3886: Deleting consensus metadata
I20260925 15:28:28.521030 17566 catalog_manager.cc:4841] TS 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): tablet 8e6f51d1ff24417ba63f8c2763b49f0b (table test-table [id=177abfe8e23f46d1b8c284b9a933d627]) successfully deleted
I20260925 15:28:28.521175 17565 catalog_manager.cc:4841] TS 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): tablet 8e6f51d1ff24417ba63f8c2763b49f0b (table test-table [id=177abfe8e23f46d1b8c284b9a933d627]) successfully deleted
W20260925 15:28:28.521190 17567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:39311 (error 108)
I20260925 15:28:28.521703 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17775
I20260925 15:28:28.521883 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:28.521730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:39311 (local address 127.0.0.1:51664)
0925 15:28:28.521799 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:28.521838 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:39311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:28:28.522755 17567 catalog_manager.cc:4584] TS 193bc2deeaf342babe54a2a3edcbe7fa (127.1.163.65:39311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e6f51d1ff24417ba63f8c2763b49f0b: Network error: Client connection negotiation failed: client connection to 127.1.163.65:39311: connect: Connection refused (error 111)
I20260925 15:28:28.527475 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17910
I20260925 15:28:28.532368 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 17548
I20260925 15:28:48.538254 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--webserver_interface=127.1.163.126
--webserver_port=44675
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:48.624848 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:48.625064 18070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:48.625087 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:48.626330 18070 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:28:48.626372 18070 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:28:48.626385 18070 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:28:48.626396 18070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:28:48.626410 18070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:48.626485 18070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:48.626516 18070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:28:48.626541 18070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:28:48.628123 18070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:36167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:48.628409 18070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:48.628705 18070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:48.636592 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:48.636592 18078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:48.636592 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:48.637282 18070 server_base.cc:1034] running on GCE node
I20260925 15:28:48.637636 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:48.638058 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:48.639266 18070 hybrid_clock.cc:648] HybridClock initialized: now 1790350128639246 us; error 43 us; skew 500 ppm
Sep 25 15:28:48 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:48.650138 18070 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:28:48.650966 18070 webserver.cc:460] Webserver started at http://127.1.163.126:44675/ using document root <none> and password file <none>
I20260925 15:28:48.651400 18070 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:48.651485 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:48.653596 18070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:28:48.656474 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:48.656767 18070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:28:48.656886 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "1ba9d19c66c74c9496a7b3da228f5406"
format_stamp: "Formatted at 2026-09-25 15:28:06 on dist-test-slave-hvv5"
server_key: "146f9f0ea032a2f5277793edd14b5c8b"
server_key_iv: "ec0080601d7550e88e5ea8714db095dd"
server_key_version: "encryptionkey@0"
I20260925 15:28:48.657469 18070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:48.674233 18070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:48.677891 18070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:48.678067 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:48.683653 18070 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:36167
I20260925 15:28:48.683786 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:36167 every 8 connection(s)
I20260925 15:28:48.684229 18070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:28:48.686108 18138 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:28:48.686957 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18070
I20260925 15:28:48.687220 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap starting.
I20260925 15:28:48.687300 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:39311
--local_ip_for_outbound_sockets=127.1.163.65
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=34083
--webserver_interface=127.1.163.65
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:28:48.700168 18138 log.cc:824] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:48.702782 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:28:48.703127 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Bootstrap complete.
I20260925 15:28:48.705560 18138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:48.705824 18138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Initialized, Role: FOLLOWER
I20260925 15:28:48.706041 18138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:48.706140 18138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:28:48.706195 18138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:28:48.706246 18138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:28:48.707365 18138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:48.707499 18138 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba9d19c66c74c9496a7b3da228f5406; no voters:
I20260925 15:28:48.707715 18138 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:28:48.707805 18142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:28:48.707939 18142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [term 3 LEADER]: Becoming Leader. State: Replica: 1ba9d19c66c74c9496a7b3da228f5406, State: Running, Role: LEADER
I20260925 15:28:48.708003 18138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:28:48.708030 18142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } }
I20260925 15:28:48.708298 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba9d19c66c74c9496a7b3da228f5406. Latest consensus state: current_term: 3 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:48.708369 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:48.708330 18144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1ba9d19c66c74c9496a7b3da228f5406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba9d19c66c74c9496a7b3da228f5406" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 36167 } } }
I20260925 15:28:48.709060 18152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:28:48.709098 18144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406 [sys.catalog]: This master's current role is: LEADER
I20260925 15:28:48.709789 18152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:28:48.709924 18152 catalog_manager.cc:1259] Loaded cluster ID: b7ee3d5b55dc4441b7f56e641299c522
I20260925 15:28:48.709960 18152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:28:48.711393 18152 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:28:48.711632 18152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Loaded TSK: 0
I20260925 15:28:48.711680 18152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ba9d19c66c74c9496a7b3da228f5406: Loaded TSK: 1
F20260925 15:28:48.712037 18152 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2ee76959ad google::LogMessage::Fail() at ??:0
@ 0x7f2ee769888a google::LogMessage::SendToLog() at ??:0
@ 0x7f2ee76954de google::LogMessage::Flush() at ??:0
@ 0x7f2ee7696a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2ee9e02104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2ee9dffd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2eef5ef1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2eef5ef48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2eef5ef594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2eef5b5c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2eef5bfad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2ee95630d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2ee9558954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2eea3bc6db start_thread at ??:0
@ 0x7f2ee619361f clone at ??:0
W20260925 15:28:48.801097 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:48.801366 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:48.801409 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:48.803298 18140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:48.803401 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:48.803467 18140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:28:48.805294 18140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:39311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=34083
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:48.805605 18140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:48.805863 18140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:48.811931 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:48.811961 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:48.811960 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:48.812156 18140 server_base.cc:1034] running on GCE node
I20260925 15:28:48.812414 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:48.812788 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:48.813994 18140 hybrid_clock.cc:648] HybridClock initialized: now 1790350128813971 us; error 37 us; skew 500 ppm
Sep 25 15:28:48 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:48.817543 18140 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:28:48.818095 18140 webserver.cc:460] Webserver started at http://127.1.163.65:34083/ using document root <none> and password file <none>
I20260925 15:28:48.818377 18140 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:48.818457 18140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:48.820094 18140 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:28:48.821039 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:48.821238 18140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:28:48.821378 18140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "193bc2deeaf342babe54a2a3edcbe7fa"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f1a430c04de8f5a59d5eca0baaba9426"
server_key_iv: "0cd6f75f72595f41e62f9ecf95c1af3f"
server_key_version: "encryptionkey@0"
I20260925 15:28:48.821758 18140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:48.840986 18140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:48.843484 18140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:48.843663 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:48.843918 18140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:48.844406 18196 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:28:48.846465 18140 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:28:48.846550 18140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:28:48.846598 18140 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:28:48.847328 18140 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:28:48.847385 18140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:28:48.847468 18196 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap starting.
I20260925 15:28:48.849483 18196 log.cc:824] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:28:48.852840 18196 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:28:48.853188 18196 tablet_bootstrap.cc:492] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Bootstrap complete.
I20260925 15:28:48.853353 18196 ts_tablet_manager.cc:1397] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 15:28:48.855202 18140 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:39311
I20260925 15:28:48.855233 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:39311 every 8 connection(s)
I20260925 15:28:48.855168 18196 raft_consensus.cc:348] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:48.855496 18196 raft_consensus.cc:729] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193bc2deeaf342babe54a2a3edcbe7fa, State: Initialized, Role: FOLLOWER
I20260925 15:28:48.855607 18140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:28:48.855681 18196 consensus_queue.cc:260] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:48.855991 18196 ts_tablet_manager.cc:1428] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:28:48.860337 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18140
I20260925 15:28:48.860703 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:43097
--local_ip_for_outbound_sockets=127.1.163.66
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=35677
--webserver_interface=127.1.163.66
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:48.932663 18308 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:28:48.856755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:41883)
0925 15:28:48.856935 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.856984 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:48.857149 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:48.932395 (+ 75246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
W20260925 15:28:48.932999 18305 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36167 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:36167: Client connection negotiation failed: client connection to 127.1.163.126:36167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:28:48.933383 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:48.933238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:36231)
0925 15:28:48.933299 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.933352 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:28:48.933785 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:48.933675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:46645)
0925 15:28:48.933728 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.933759 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:28:48.933943 18305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:28:48.945739 18309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:48.945964 18309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:48.945990 18309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:48.947232 18309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:48.947285 18309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:48.947314 18309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:28:48.948933 18309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:43097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=35677
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:48.949201 18309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:48.949445 18309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:48.954881 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:48.954945 18309 server_base.cc:1034] running on GCE node
W20260925 15:28:48.954924 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:48.954902 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:48.955339 18309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:48.955560 18309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:48.956712 18309 hybrid_clock.cc:648] HybridClock initialized: now 1790350128956680 us; error 45 us; skew 500 ppm
Sep 25 15:28:48 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:48.959721 18309 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:28:48.960258 18309 webserver.cc:460] Webserver started at http://127.1.163.66:35677/ using document root <none> and password file <none>
I20260925 15:28:48.960587 18309 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:48.960688 18309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:48.962272 18309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:48.963018 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:48.963235 18309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:48.963358 18309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "48d2a2aa986a44d288aa35bac2e79f2c"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "f390ed59a862b1e687bf85fad5fbdd3f"
server_key_iv: "cf01001a7c41f8fcb8e912456e155445"
server_key_version: "encryptionkey@0"
I20260925 15:28:48.963816 18309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:48.978219 18309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:48.980818 18309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:48.980998 18309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:48.981377 18309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:48.981770 18309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:48.981886 18309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:48.981931 18309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:48.981979 18309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:48.988641 18309 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:43097
I20260925 15:28:48.988722 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:43097 every 8 connection(s)
I20260925 15:28:48.989152 18309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:28:48.990361 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:48.989911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:54697)
0925 15:28:48.990107 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.990204 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
W20260925 15:28:48.990800 18438 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36167 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:36167: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
I20260925 15:28:48.991117 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:48.991020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:41905)
0925 15:28:48.991071 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.991099 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:28:48.991425 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:48.991375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:45337)
0925 15:28:48.991396 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:48.991414 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 15:28:48.991545 18438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:28:48.996753 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18309
I20260925 15:28:48.997141 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:34309
--local_ip_for_outbound_sockets=127.1.163.67
--tserver_master_addrs=127.1.163.126:36167
--webserver_port=34863
--webserver_interface=127.1.163.67
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:38147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:28:49.083346 18441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:28:49.083564 18441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:28:49.083590 18441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:28:49.084873 18441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:28:49.084920 18441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:28:49.084949 18441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:28:49.086474 18441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:38147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:34309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=34863
--tserver_master_addrs=127.1.163.126:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:28:49.086735 18441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:28:49.087002 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:28:49.092386 18446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:28:49.092358 18449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:49.092481 18441 server_base.cc:1034] running on GCE node
W20260925 15:28:49.092378 18447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:28:49.092907 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:28:49.093179 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:28:49.094350 18441 hybrid_clock.cc:648] HybridClock initialized: now 1790350129094345 us; error 38 us; skew 500 ppm
Sep 25 15:28:49 dist-test-slave-hvv5 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350129, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:28:49.096957 18441 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:28:49.097436 18441 webserver.cc:460] Webserver started at http://127.1.163.67:34863/ using document root <none> and password file <none>
I20260925 15:28:49.097668 18441 fs_manager.cc:359] Metadata directory not provided
I20260925 15:28:49.097783 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:28:49.099293 18441 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:49.100024 18456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:49.100248 18441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:28:49.100361 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "7bd776111e874b26a8b947198e6b3886"
format_stamp: "Formatted at 2026-09-25 15:28:07 on dist-test-slave-hvv5"
server_key: "bfbbae568bed143cb1522cd69b84446a"
server_key_iv: "f4cde867e8ae59aa2680a10e6a76e33c"
server_key_version: "encryptionkey@0"
I20260925 15:28:49.100836 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:28:49.114164 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:28:49.116618 18441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:28:49.116784 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:28:49.117030 18441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:28:49.117406 18441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:28:49.117462 18441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:49.117506 18441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:28:49.117533 18441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:28:49.123520 18441 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:34309
I20260925 15:28:49.123601 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:34309 every 8 connection(s)
I20260925 15:28:49.123971 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:28:49.125241 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:49.124846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:43227)
0925 15:28:49.125020 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:49.125094 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
W20260925 15:28:49.125743 18570 heartbeater.cc:644] Failed to heartbeat to 127.1.163.126:36167 (0 consecutive failures): Network error: Failed to ping master at 127.1.163.126:36167: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
I20260925 15:28:49.126071 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:49.125975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:52801)
0925 15:28:49.126026 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:49.126053 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:28:49.126426 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:49.126350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:41005)
0925 15:28:49.126371 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:49.126411 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 15:28:49.126573 18570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:28:49.132896 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18441
I20260925 15:28:49.934780 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:49.934461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:34349)
0925 15:28:49.934611 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:49.934674 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260925 15:28:49.992280 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:49.991994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:47669)
0925 15:28:49.992160 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:49.992221 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260925 15:28:50.127197 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.126950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:51403)
0925 15:28:50.127099 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:50.127142 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260925 15:28:50.384671 18577 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:50.384796 18577 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:50.385133 18577 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
Sep 25 15:28:50 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.67@KRBTEST.COM
Sep 25 15:28:50 dist-test-slave-hvv5 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350128, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.66@KRBTEST.COM
I20260925 15:28:50.397030 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.385437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:36399 (local address 127.1.163.66:43097)
0925 15:28:50.385606 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:50.385618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:50.385653 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:28:50.386246 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:50.386249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:50.386307 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:50.386476 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:50.387138 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.388082 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389238 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.389655 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:50.392607 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:50.392612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:50.392612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:50.392625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:50.394674 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395449 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:50.395452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:50.395453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:50.395595 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395937 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:50.395938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:50.395938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:50.396264 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:50.396361 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:50.396764 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 15:28:50.396912 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260925 15:28:50.397055 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.385459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:46499 (local address 127.1.163.67:34309)
0925 15:28:50.385608 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:28:50.385618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:28:50.385677 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:28:50.386246 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:28:50.386249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:28:50.386307 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:28:50.386476 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:28:50.387329 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.388239 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389229 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.389648 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389766 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:50.392252 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:28:50.392257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:28:50.392257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:28:50.392272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:28:50.394660 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395449 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:50.395452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:50.395453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:50.395595 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395937 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:28:50.395938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:28:50.395938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:28:50.396264 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:28:50.396353 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:28:50.396764 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:28:50.396953 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":50,"threads_started":1}
I20260925 15:28:50.397113 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.385314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34309 (local address 127.1.163.65:46499)
0925 15:28:50.385613 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:50.385623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:50.385684 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:50.386603 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:50.386605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:50.387144 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:50.387217 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:50.387219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.388343 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.388344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:50.389139 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:50.389141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.389742 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:50.389886 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:50.390570 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:50.390575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:50.392118 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:50.394840 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:50.394841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:50.394841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:50.395341 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:50.395682 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:50.395683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:50.395860 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:50.396350 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:50.396359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:50.396557 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:28:50.396696 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:28:50.396986 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":134,"thread_start_us":62,"threads_started":1}
I20260925 15:28:50.397790 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.385314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:43097 (local address 127.1.163.65:36399)
0925 15:28:50.385399 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:50.385412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:28:50.385466 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:28:50.386518 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:28:50.386520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:28:50.386798 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:28:50.387004 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:50.387010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.388197 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.388198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:50.389140 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:28:50.389142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:28:50.389766 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:28:50.389767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:28:50.389887 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:28:50.390514 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:28:50.390523 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:28:50.392464 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:28:50.394783 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:50.394793 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:50.394796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:50.395341 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:50.395682 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:28:50.395683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:28:50.395683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:28:50.395854 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:28:50.396350 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:28:50.396359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:28:50.396557 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:28:50.396696 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:28:50.396925 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":49}
W20260925 15:28:50.397930 18190 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
W20260925 15:28:50.398651 18191 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:50.398706 18191 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa; no voters: 48d2a2aa986a44d288aa35bac2e79f2c, 7bd776111e874b26a8b947198e6b3886
I20260925 15:28:50.398847 18577 raft_consensus.cc:2726] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:28:50.935719 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.935450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:33455)
0925 15:28:50.935588 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:50.935653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260925 15:28:50.993188 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:50.992939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:48635)
0925 15:28:50.993099 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:50.993143 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260925 15:28:51.127804 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:51.127614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:38277)
0925 15:28:51.127727 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:51.127763 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260925 15:28:51.936414 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:51.936148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:47237)
0925 15:28:51.936312 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:51.936358 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260925 15:28:51.994050 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:51.993759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:48697)
0925 15:28:51.993914 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:51.993959 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260925 15:28:52.128580 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:52.128295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:56485)
0925 15:28:52.128452 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:52.128488 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 15:28:52.465188 18587 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:52.465296 18587 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:52.465523 18587 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
W20260925 15:28:52.465950 18190 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
W20260925 15:28:52.466092 18191 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:52.466163 18191 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa; no voters: 48d2a2aa986a44d288aa35bac2e79f2c, 7bd776111e874b26a8b947198e6b3886
I20260925 15:28:52.466236 18587 raft_consensus.cc:2726] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:28:52.937198 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:52.936955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:54419)
0925 15:28:52.937093 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:52.937133 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260925 15:28:52.994927 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:52.994652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:45277)
0925 15:28:52.994820 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:52.994875 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260925 15:28:53.129401 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:53.129143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:38721)
0925 15:28:53.129299 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:53.129343 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260925 15:28:53.938139 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:53.937856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:32955)
0925 15:28:53.938042 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:53.938089 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260925 15:28:53.995909 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:53.995570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:38473)
0925 15:28:53.995754 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:53.995800 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260925 15:28:54.130183 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:54.129926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:35833)
0925 15:28:54.130085 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:54.130129 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260925 15:28:54.939146 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:54.938842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:33593)
0925 15:28:54.939046 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:54.939094 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260925 15:28:54.997033 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:54.996696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:45373)
0925 15:28:54.996935 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:54.996983 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260925 15:28:55.131222 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:55.130892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:46539)
0925 15:28:55.131126 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:55.131173 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260925 15:28:55.223421 18597 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:55.223660 18597 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:55.223918 18597 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
W20260925 15:28:55.224466 18191 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
W20260925 15:28:55.224584 18190 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:55.224689 18190 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa; no voters: 48d2a2aa986a44d288aa35bac2e79f2c, 7bd776111e874b26a8b947198e6b3886
I20260925 15:28:55.224817 18597 raft_consensus.cc:2726] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:28:55.940124 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:55.939804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:36533)
0925 15:28:55.940020 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:55.940076 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":114,"threads_started":1}
I20260925 15:28:55.997895 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:55.997630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:47671)
0925 15:28:55.997793 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:55.997848 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260925 15:28:56.132200 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:56.131936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:48113)
0925 15:28:56.132095 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:56.132148 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260925 15:28:56.940972 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:56.940721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:45781)
0925 15:28:56.940875 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:56.940925 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260925 15:28:56.998726 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:56.998500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:60597)
0925 15:28:56.998634 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:56.998677 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260925 15:28:57.133069 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:57.132829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:38095)
0925 15:28:57.132974 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:57.133020 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260925 15:28:57.941919 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:57.941485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:57585)
0925 15:28:57.941791 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:57.941851 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260925 15:28:57.999595 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:57.999353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:43493)
0925 15:28:57.999501 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:57.999544 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260925 15:28:58.134037 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:58.133643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:46705)
0925 15:28:58.133880 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:58.133929 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
I20260925 15:28:58.492722 18607 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:28:58.492830 18607 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:28:58.493059 18607 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
W20260925 15:28:58.493512 18190 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
W20260925 15:28:58.493594 18191 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:28:58.493656 18191 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa; no voters: 48d2a2aa986a44d288aa35bac2e79f2c, 7bd776111e874b26a8b947198e6b3886
I20260925 15:28:58.493746 18607 raft_consensus.cc:2726] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:28:58.942843 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:58.942556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:47609)
0925 15:28:58.942730 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:58.942776 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260925 15:28:59.000397 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:59.000117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:58557)
0925 15:28:59.000285 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:59.000346 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260925 15:28:59.134917 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:59.134645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:37011)
0925 15:28:59.134820 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:59.134870 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260925 15:28:59.943940 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:28:59.943626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:42991)
0925 15:28:59.943841 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:28:59.943890 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260925 15:29:00.001482 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:00.001081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:50325)
0925 15:29:00.001338 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:00.001387 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260925 15:29:00.135859 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:00.135607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:53773)
0925 15:29:00.135761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:00.135809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260925 15:29:00.293074 18614 raft_consensus.cc:482] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:00.293195 18614 raft_consensus.cc:504] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd776111e874b26a8b947198e6b3886" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34309 } } peers { permanent_uuid: "48d2a2aa986a44d288aa35bac2e79f2c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 43097 } } peers { permanent_uuid: "193bc2deeaf342babe54a2a3edcbe7fa" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 39311 } }
I20260925 15:29:00.293406 18614 leader_election.cc:288] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309), 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097)
W20260925 15:29:00.293964 18191 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48d2a2aa986a44d288aa35bac2e79f2c (127.1.163.66:43097): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
W20260925 15:29:00.294124 18190 leader_election.cc:341] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bd776111e874b26a8b947198e6b3886 (127.1.163.67:34309): Not found: Tablet not found: 8e6f51d1ff24417ba63f8c2763b49f0b
I20260925 15:29:00.294202 18190 leader_election.cc:302] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 193bc2deeaf342babe54a2a3edcbe7fa; no voters: 48d2a2aa986a44d288aa35bac2e79f2c, 7bd776111e874b26a8b947198e6b3886
I20260925 15:29:00.294341 18614 raft_consensus.cc:2726] T 8e6f51d1ff24417ba63f8c2763b49f0b P 193bc2deeaf342babe54a2a3edcbe7fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:29:00.944973 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:00.944692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:43455)
0925 15:29:00.944876 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:00.944924 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260925 15:29:01.002348 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:01.002088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:41955)
0925 15:29:01.002252 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:01.002301 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 15:29:01.136922 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:01.136642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:52773)
0925 15:29:01.136827 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:01.136872 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260925 15:29:01.945974 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:01.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:38751)
0925 15:29:01.945877 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:01.945924 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260925 15:29:02.003582 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:02.003255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:42587)
0925 15:29:02.003477 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:02.003531 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260925 15:29:02.138063 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:02.137700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:44249)
0925 15:29:02.137960 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:02.138013 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260925 15:29:02.946832 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:02.946572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:39191)
0925 15:29:02.946740 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:02.946783 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260925 15:29:03.004633 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:03.004325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:37329)
0925 15:29:03.004480 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:03.004527 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260925 15:29:03.138983 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:03.138730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:44167)
0925 15:29:03.138887 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:03.138935 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260925 15:29:03.947654 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:03.947408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.65:53345)
0925 15:29:03.947569 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:03.947607 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260925 15:29:04.006110 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:04.005808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.66:58853)
0925 15:29:04.006015 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:04.006061 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260925 15:29:04.139323 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18140
I20260925 15:29:04.140303 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:04.139710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:36167 (local address 127.1.163.67:52299)
0925 15:29:04.140202 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:04.140245 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.126:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":53,"threads_started":1}
I20260925 15:29:04.145028 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18309
I20260925 15:29:04.149676 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18441
2026-09-25T15:29:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58488 ms)
[----------] 30 tests from SecurityITest (240120 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:04 dist-test-slave-hvv5 krb5kdc[18629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:04 dist-test-slave-hvv5 krb5kdc[18629](info): set up 2 sockets
Sep 25 15:29:04 dist-test-slave-hvv5 krb5kdc[18629](info): commencing operation
krb5kdc: starting...
W20260925 15:29:05.228514 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:05.368741 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38977
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:05.451901 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:05.452147 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:05.452198 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:05.453501 18642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:05.453550 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:05.453565 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:05.453578 18642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:05.453590 18642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:05.455260 18642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:05.455566 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:05.455821 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:05.461458 18650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.461433 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.461426 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:05.461489 18642 server_base.cc:1034] running on GCE node
I20260925 15:29:05.462081 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:05.462394 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:05.463572 18642 hybrid_clock.cc:648] HybridClock initialized: now 1790350145463568 us; error 39 us; skew 500 ppm
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:05.466217 18642 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:05.466792 18642 webserver.cc:460] Webserver started at http://127.1.163.126:46169/ using document root <none> and password file <none>
I20260925 15:29:05.467199 18642 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:05.467280 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:05.467435 18642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:05.468873 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "ccd94e6eccfc4e38b1c33900b7d96457"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0ffed64babe71fe92061dfc5bf60710f"
server_key_iv: "921cbe9eb081b54476b225a634948642"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.469269 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "ccd94e6eccfc4e38b1c33900b7d96457"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0ffed64babe71fe92061dfc5bf60710f"
server_key_iv: "921cbe9eb081b54476b225a634948642"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.470728 18642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:05.471649 18657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.471897 18642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:05.472014 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "ccd94e6eccfc4e38b1c33900b7d96457"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0ffed64babe71fe92061dfc5bf60710f"
server_key_iv: "921cbe9eb081b54476b225a634948642"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.472097 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:05.487540 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:05.490092 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:05.490216 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:05.494308 18642 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:38977
I20260925 15:29:05.494401 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:38977 every 8 connection(s)
I20260925 15:29:05.494767 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:05.495810 18710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:05.498394 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: Bootstrap starting.
I20260925 15:29:05.499029 18710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:05.499346 18710 log.cc:824] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:05.500202 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: No bootstrap required, opened a new log
I20260925 15:29:05.502148 18710 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } }
I20260925 15:29:05.502332 18710 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:05.502358 18710 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd94e6eccfc4e38b1c33900b7d96457, State: Initialized, Role: FOLLOWER
I20260925 15:29:05.502533 18710 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } }
I20260925 15:29:05.502625 18710 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:05.502653 18710 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:05.502719 18710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:05.503433 18710 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } }
I20260925 15:29:05.503537 18710 leader_election.cc:302] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd94e6eccfc4e38b1c33900b7d96457; no voters:
I20260925 15:29:05.503750 18710 leader_election.cc:288] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:05.503994 18713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:05.504179 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18642
I20260925 15:29:05.504245 18713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [term 1 LEADER]: Becoming Leader. State: Replica: ccd94e6eccfc4e38b1c33900b7d96457, State: Running, Role: LEADER
I20260925 15:29:05.504292 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:05.504364 18713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } }
I20260925 15:29:05.504417 18710 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:05.504597 1677 external_mini_cluster.cc:1428] Setting key 25d4fc6181cd35c30a4bf5ef954a5b25
I20260925 15:29:05.505785 18715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd94e6eccfc4e38b1c33900b7d96457. Latest consensus state: current_term: 1 leader_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } } }
I20260925 15:29:05.505796 18714 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd94e6eccfc4e38b1c33900b7d96457" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38977 } } }
I20260925 15:29:05.505908 18715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:05.505939 18714 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457 [sys.catalog]: This master's current role is: LEADER
W20260925 15:29:05.506677 18731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:05.506737 18731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:05.506865 18728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:05.507053 18728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:05.508932 18728 catalog_manager.cc:1347] Generated new cluster ID: 85950ac419014c62b3353859a173e72f
I20260925 15:29:05.508997 18728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:05.516880 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:05.505117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.1.163.126:38977)
0925 15:29:05.505495 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:05.505506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:05.505544 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:29:05.505729 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:05.505739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:05.505812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:05.506527 (+ 715us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:05.506844 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.508115 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.509369 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.509800 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.509947 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:05.512333 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:05.512338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:05.512339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:05.512352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:05.515099 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.515501 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.515505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.515506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.515633 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.515857 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.515858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.515858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.516131 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:05.516349 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:05.516435 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:29:05.516618 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":80,"threads_started":1}
I20260925 15:29:05.534484 18728 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:05.535266 18728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:05.539979 18728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: Generated new TSK 0
I20260925 15:29:05.540203 18728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:29:05.573413 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:05.674225 18736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:05.674532 18736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:05.674577 18736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:05.675942 18736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:05.676005 18736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:05.676052 18736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:05.677580 18736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:05.677857 18736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:05.678102 18736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:05.683537 18742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.683531 18741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.683531 18744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:05.683665 18736 server_base.cc:1034] running on GCE node
I20260925 15:29:05.684002 18736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:05.684305 18736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:05.685475 18736 hybrid_clock.cc:648] HybridClock initialized: now 1790350145685444 us; error 52 us; skew 500 ppm
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:05.688375 18736 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:29:05.688974 18736 webserver.cc:460] Webserver started at http://127.1.163.65:36719/ using document root <none> and password file <none>
I20260925 15:29:05.689248 18736 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:05.689329 18736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:05.689492 18736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:05.690727 18736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "57874a9004194f888b68b896d7c8d838"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0886c17b3235ae37c86c9e38d80e610c"
server_key_iv: "4b6639d0e7c4e8120647345a4d55567f"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.691138 18736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "57874a9004194f888b68b896d7c8d838"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0886c17b3235ae37c86c9e38d80e610c"
server_key_iv: "4b6639d0e7c4e8120647345a4d55567f"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.692684 18736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:05.693554 18751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.693921 18736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:05.694062 18736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "57874a9004194f888b68b896d7c8d838"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "0886c17b3235ae37c86c9e38d80e610c"
server_key_iv: "4b6639d0e7c4e8120647345a4d55567f"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.694149 18736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:05.713864 18736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:05.716069 18736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:05.716251 18736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:05.716670 18736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:05.717128 18736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:05.717191 18736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.717236 18736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:05.717267 18736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.723819 18736 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:33005
I20260925 15:29:05.723862 18864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:33005 every 8 connection(s)
I20260925 15:29:05.724268 18736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:29:05.730417 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18736
I20260925 15:29:05.730549 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:05.730777 1677 external_mini_cluster.cc:1428] Setting key 22aceb51181f841de246b412f2244b26
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:05.734952 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:05.725122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:49181 (local address 127.1.163.126:38977)
0925 15:29:05.725269 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:05.725272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:05.725475 (+ 203us) server_negotiation.cc:407] Connection header received
0925 15:29:05.726032 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:05.726033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:05.726052 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:05.726107 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:05.727181 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.727708 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.728848 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.729064 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.729156 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:05.731825 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:05.731826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:05.731827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:05.731842 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:05.733431 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.733918 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.733920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.733920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.733983 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.734257 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.734258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.734258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.734389 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:05.734468 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:05.734766 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:29:05.734859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 15:29:05.735179 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:05.725071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38977 (local address 127.1.163.65:49181)
0925 15:29:05.725305 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:05.725367 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:05.725905 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:05.726187 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:05.726191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:05.726808 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:05.727063 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:05.727068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.727821 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.727822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:05.728750 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:05.728753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.729172 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.729173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:05.729310 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:05.729957 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:05.729967 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:05.731698 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:05.733556 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:05.733565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:05.733568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:05.733825 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:05.734060 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:05.734061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:05.734061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:05.734181 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:05.734458 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:05.734466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:05.734619 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 15:29:05.734736 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:29:05.734913 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260925 15:29:05.735723 18865 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38977
I20260925 15:29:05.735852 18865 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:05.736040 18865 heartbeater.cc:511] Master 127.1.163.126:38977 requested a full tablet report, sending...
I20260925 15:29:05.736696 18672 ts_manager.cc:194] Registered new tserver with Master: 57874a9004194f888b68b896d7c8d838 (127.1.163.65:33005)
I20260925 15:29:05.737344 18672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:49181
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:29:05.761664 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:05.846210 18870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:05.846514 18870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:05.846563 18870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:05.847833 18870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:05.847886 18870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:05.847913 18870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:05.849453 18870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:05.849675 18870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:05.849874 18870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:05.855495 18878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.855495 18875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:05.855531 18876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:05.855574 18870 server_base.cc:1034] running on GCE node
I20260925 15:29:05.855971 18870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:05.856405 18870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:05.857635 18870 hybrid_clock.cc:648] HybridClock initialized: now 1790350145857549 us; error 135 us; skew 500 ppm
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:05.861024 18870 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:29:05.861748 18870 webserver.cc:460] Webserver started at http://127.1.163.66:33827/ using document root <none> and password file <none>
I20260925 15:29:05.862015 18870 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:05.862090 18870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:05.862246 18870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:05.863404 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "7853bae086514f24b6f02af165e9d9ab"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "fd4b37b756ee48316bdf166bbfe0b215"
server_key_iv: "6514419d7a0f5e8405e053693b622c61"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.863791 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "7853bae086514f24b6f02af165e9d9ab"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "fd4b37b756ee48316bdf166bbfe0b215"
server_key_iv: "6514419d7a0f5e8405e053693b622c61"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.865319 18870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:05.866356 18885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.866690 18870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:05.866886 18870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "7853bae086514f24b6f02af165e9d9ab"
format_stamp: "Formatted at 2026-09-25 15:29:05 on dist-test-slave-hvv5"
server_key: "fd4b37b756ee48316bdf166bbfe0b215"
server_key_iv: "6514419d7a0f5e8405e053693b622c61"
server_key_version: "encryptionkey@0"
I20260925 15:29:05.866981 18870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:05.878918 18870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:05.881592 18870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:05.881759 18870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:05.882109 18870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:05.882465 18870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:05.882499 18870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.882522 18870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:05.882591 18870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:05.889400 18870 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39031
I20260925 15:29:05.889479 18998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39031 every 8 connection(s)
I20260925 15:29:05.889806 18870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:29:05 dist-test-slave-hvv5 krb5kdc[18629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350145, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:05.897424 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 18870
I20260925 15:29:05.897562 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:05.897816 1677 external_mini_cluster.cc:1428] Setting key d7611d9d7cc4621b41f53c4195ca983f
I20260925 15:29:05.900820 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:05.890691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:58083 (local address 127.1.163.126:38977)
0925 15:29:05.890767 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:05.890769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:05.890988 (+ 219us) server_negotiation.cc:407] Connection header received
0925 15:29:05.891489 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:05.891491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:05.891502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:05.891575 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:05.892512 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.893137 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.894143 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.894346 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.894415 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:05.896957 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:05.896959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:05.896959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:05.896967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:05.898960 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.899568 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.899569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.899570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.899630 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:05.899978 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:05.899980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:05.899980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:05.900162 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:05.900248 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:05.900572 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 15:29:05.900682 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:29:05.900895 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:05.890677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38977 (local address 127.1.163.66:58083)
0925 15:29:05.890841 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:05.890884 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:05.891369 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:05.891630 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:05.891633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:05.892206 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:05.892410 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:05.892416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.893221 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.893222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:05.894032 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:05.894035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:05.894443 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:05.894444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:05.894552 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:05.895077 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:05.895086 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:05.896843 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:05.899050 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:05.899059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:05.899062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:05.899478 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:05.899709 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:05.899710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:05.899711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:05.899864 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:05.900248 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:05.900254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:05.900394 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 15:29:05.900484 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:29:05.900696 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260925 15:29:05.901446 18999 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38977
I20260925 15:29:05.901544 18999 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:05.901753 18999 heartbeater.cc:511] Master 127.1.163.126:38977 requested a full tablet report, sending...
I20260925 15:29:05.902215 18672 ts_manager.cc:194] Registered new tserver with Master: 7853bae086514f24b6f02af165e9d9ab (127.1.163.66:39031)
I20260925 15:29:05.902733 18672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:58083
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:29:05.932435 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:36523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:06.019176 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:06.019438 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:06.019467 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:06.020772 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:06.020823 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:06.020854 19004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:06.022672 19004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:36523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:06.023002 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:06.023350 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:06.028728 19010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.028685 19009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.028790 19004 server_base.cc:1034] running on GCE node
W20260925 15:29:06.028703 19012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.029210 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:06.029488 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:06.030699 19004 hybrid_clock.cc:648] HybridClock initialized: now 1790350146030675 us; error 40 us; skew 500 ppm
Sep 25 15:29:06 dist-test-slave-hvv5 krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:06.033739 19004 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:29:06.034271 19004 webserver.cc:460] Webserver started at http://127.1.163.67:33987/ using document root <none> and password file <none>
I20260925 15:29:06.034519 19004 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:06.034593 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:06.034747 19004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:06.035943 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "fb2e1b3dea254e6f8ca21480686dbd29"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "a8c2c396c4e39d93c0f4b51a8a2ba682"
server_key_iv: "bdccd3e297e9ed0ec5a9e9728c675306"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.036361 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "fb2e1b3dea254e6f8ca21480686dbd29"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "a8c2c396c4e39d93c0f4b51a8a2ba682"
server_key_iv: "bdccd3e297e9ed0ec5a9e9728c675306"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.038539 19004 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 15:29:06.039613 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.039974 19004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:06.040196 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "fb2e1b3dea254e6f8ca21480686dbd29"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "a8c2c396c4e39d93c0f4b51a8a2ba682"
server_key_iv: "bdccd3e297e9ed0ec5a9e9728c675306"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.040331 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:06.052834 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:06.055226 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:06.055426 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:06.055722 19004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:06.056070 19004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:06.056128 19004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.056177 19004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:06.056209 19004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.064442 19004 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:36549
I20260925 15:29:06.064587 19132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:36549 every 8 connection(s)
I20260925 15:29:06.064924 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:29:06.067968 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19004
I20260925 15:29:06.068185 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:06.068356 1677 external_mini_cluster.cc:1428] Setting key 82e8e9bceec9b7b9eade9f30a0018ca8
Sep 25 15:29:06 dist-test-slave-hvv5 krb5kdc[18629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:06.075783 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.065872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:42673 (local address 127.1.163.126:38977)
0925 15:29:06.065924 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.065926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.066080 (+ 154us) server_negotiation.cc:407] Connection header received
0925 15:29:06.067116 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.067118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.067134 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.067207 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.068113 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.068825 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.069937 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.070152 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.070230 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.072600 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.072601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.072602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:06.072610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.074098 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:06.074643 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:06.074645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:06.074646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:06.074717 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:06.075008 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:06.075009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:06.075009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:06.075136 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.075233 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.075530 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.075653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:29:06.076028 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.065735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38977 (local address 127.1.163.67:42673)
0925 15:29:06.065914 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.065960 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.066926 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.067411 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.067416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.067784 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:06.067994 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.067998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.068925 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.068927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.069845 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.069848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.070251 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.070252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.070384 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.070992 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:06.071002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:06.072465 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:06.074222 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:06.074230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:06.074233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:06.074552 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:06.074799 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:06.074800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:06.074801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:06.074935 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:06.075227 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:06.075233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:06.075386 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.075506 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.075692 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 15:29:06.076629 19133 heartbeater.cc:347] Connected to a master server at 127.1.163.126:38977
I20260925 15:29:06.076738 19133 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:06.076941 19133 heartbeater.cc:511] Master 127.1.163.126:38977 requested a full tablet report, sending...
I20260925 15:29:06.077410 18672 ts_manager.cc:194] Registered new tserver with Master: fb2e1b3dea254e6f8ca21480686dbd29 (127.1.163.67:36549)
I20260925 15:29:06.077849 18672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:42673
I20260925 15:29:06.080605 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:06.089679 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.082044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.1.163.126:38977)
0925 15:29:06.082132 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.082134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.082185 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:29:06.082240 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.082241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.082250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.082300 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.082954 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.083462 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.084334 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.084645 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.084710 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.086028 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.086030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.086030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:06.086038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.087602 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:06.088039 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:06.088040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:06.088040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:06.088105 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:06.088474 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:06.088475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:06.088476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:06.088740 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.088961 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.089301 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.089388 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:29:06.091879 18672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:06.093752 18672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:06.102238 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.1.163.66:39031)
0925 15:29:06.097954 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.097959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.098272 (+ 313us) server_negotiation.cc:407] Connection header received
0925 15:29:06.098329 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.098332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.098350 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.098466 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.098650 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.099448 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.100969 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101739 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101829 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.101891 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.102088 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.102137 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260925 15:29:06.102273 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33005 (local address 127.0.0.1:39810)
0925 15:29:06.097499 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.097549 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.097630 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.098548 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.098549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.098552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.098615 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.098617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.099991 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.099992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.101098 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.101101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101973 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.102032 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.102035 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.102084 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.102119 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":29,"thread_start_us":156,"threads_started":1}
I20260925 15:29:06.102319 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.1.163.65:33005)
0925 15:29:06.098382 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.098387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.098396 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:06.098435 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.098437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.098451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.098693 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.098777 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.099875 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101168 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101889 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101960 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.102017 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.102181 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.102238 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":78,"threads_started":1}
I20260925 15:29:06.102528 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39031 (local address 127.0.0.1:39656)
0925 15:29:06.098015 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.098048 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.098112 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.098486 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.098489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.098492 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.098571 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.098573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.099555 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.099556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.100861 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.100864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101931 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.102012 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.102016 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.102071 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.102126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260925 15:29:06.102159 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:36549 (local address 127.0.0.1:40736)
0925 15:29:06.098189 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.098196 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.098253 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.098559 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.098560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.098561 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.098617 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.098618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.100003 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.100005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.101004 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.101007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101786 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.101874 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.101888 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.101986 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.102053 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260925 15:29:06.103065 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.097680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.1.163.67:36549)
0925 15:29:06.097953 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.097959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.098258 (+ 299us) server_negotiation.cc:407] Connection header received
0925 15:29:06.098338 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.098340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.098358 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.098437 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.098698 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.099874 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101114 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.101668 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.101781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.101850 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.102228 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.102286 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260925 15:29:06.103068 18799 tablet_service.cc:1467] Processing CreateTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 (DEFAULT_TABLE table=test-table [id=46c1a5b7d12f462f8c9ce9e39ca78256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.103230 18933 tablet_service.cc:1467] Processing CreateTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 (DEFAULT_TABLE table=test-table [id=46c1a5b7d12f462f8c9ce9e39ca78256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.103492 18799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e734ab45c4044a57b8b2057a0a9e35a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.103588 18933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e734ab45c4044a57b8b2057a0a9e35a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.103657 19067 tablet_service.cc:1467] Processing CreateTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 (DEFAULT_TABLE table=test-table [id=46c1a5b7d12f462f8c9ce9e39ca78256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.103965 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e734ab45c4044a57b8b2057a0a9e35a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.106549 19150 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Bootstrap starting.
I20260925 15:29:06.107187 19152 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Bootstrap starting.
I20260925 15:29:06.107458 19150 tablet_bootstrap.cc:654] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.107477 19151 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Bootstrap starting.
I20260925 15:29:06.108208 19151 tablet_bootstrap.cc:654] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.108250 19152 tablet_bootstrap.cc:654] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.108234 19150 log.cc:824] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.109110 19152 log.cc:824] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.109352 19151 log.cc:824] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.109717 19150 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: No bootstrap required, opened a new log
I20260925 15:29:06.110024 19150 ts_tablet_manager.cc:1397] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:29:06.110270 19151 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: No bootstrap required, opened a new log
I20260925 15:29:06.110329 19151 ts_tablet_manager.cc:1397] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:29:06.111912 19150 raft_consensus.cc:348] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.112103 19150 raft_consensus.cc:374] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:06.112152 19150 raft_consensus.cc:729] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57874a9004194f888b68b896d7c8d838, State: Initialized, Role: FOLLOWER
I20260925 15:29:06.112121 19151 raft_consensus.cc:348] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.112282 19150 consensus_queue.cc:260] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.112360 19151 raft_consensus.cc:374] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:06.112387 19151 raft_consensus.cc:729] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2e1b3dea254e6f8ca21480686dbd29, State: Initialized, Role: FOLLOWER
I20260925 15:29:06.112557 19151 consensus_queue.cc:260] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.112654 18865 heartbeater.cc:503] Master 127.1.163.126:38977 was elected leader, sending a full tablet report...
I20260925 15:29:06.112967 19151 ts_tablet_manager.cc:1428] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:06.112660 19150 ts_tablet_manager.cc:1428] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:06.113116 19133 heartbeater.cc:503] Master 127.1.163.126:38977 was elected leader, sending a full tablet report...
I20260925 15:29:06.113924 19152 tablet_bootstrap.cc:492] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: No bootstrap required, opened a new log
I20260925 15:29:06.114073 19152 ts_tablet_manager.cc:1397] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 15:29:06.116278 19152 raft_consensus.cc:348] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.116474 19152 raft_consensus.cc:374] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:06.116559 19152 raft_consensus.cc:729] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7853bae086514f24b6f02af165e9d9ab, State: Initialized, Role: FOLLOWER
I20260925 15:29:06.116704 19152 consensus_queue.cc:260] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.117062 18999 heartbeater.cc:503] Master 127.1.163.126:38977 was elected leader, sending a full tablet report...
I20260925 15:29:06.117071 19152 ts_tablet_manager.cc:1428] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 15:29:06.140451 19000 tablet.cc:2369] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:29:06.225109 18866 tablet.cc:2369] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:06.246312 19158 raft_consensus.cc:482] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:06.246517 19158 raft_consensus.cc:504] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.247005 19158 leader_election.cc:288] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57874a9004194f888b68b896d7c8d838 (127.1.163.65:33005), fb2e1b3dea254e6f8ca21480686dbd29 (127.1.163.67:36549)
I20260925 15:29:06.251000 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.247247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:54029 (local address 127.1.163.65:33005)
0925 15:29:06.247320 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.247323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.247354 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:29:06.247385 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.247386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.247397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.247495 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.247672 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.248466 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250055 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.250626 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250698 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.250707 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.250892 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.250933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:29:06.251003 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.247175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33005 (local address 127.1.163.66:54029)
0925 15:29:06.247247 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.247260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.247314 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.247513 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.247514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.247516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.247596 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.247598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.248616 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.248618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.249957 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.249961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.250712 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.250821 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.250824 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.250877 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.250908 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":42}
I20260925 15:29:06.251044 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.247447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:44143 (local address 127.1.163.67:36549)
0925 15:29:06.247499 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.247501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.247547 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:29:06.247615 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.247616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.247624 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.247698 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.247846 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.248564 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250054 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.250619 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250698 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.250707 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.250892 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.250976 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:29:06.251010 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.247300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:36549 (local address 127.1.163.66:44143)
0925 15:29:06.247495 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.247505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.247558 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.247714 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.247714 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.247716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.247779 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.247780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.248620 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.248621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.249957 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.249961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.250712 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.250713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.250790 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.250796 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.250862 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.250907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":61,"thread_start_us":108,"threads_started":1}
I20260925 15:29:06.251487 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e734ab45c4044a57b8b2057a0a9e35a6" candidate_uuid: "7853bae086514f24b6f02af165e9d9ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" is_pre_election: true
I20260925 15:29:06.251502 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e734ab45c4044a57b8b2057a0a9e35a6" candidate_uuid: "7853bae086514f24b6f02af165e9d9ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57874a9004194f888b68b896d7c8d838" is_pre_election: true
I20260925 15:29:06.251642 19087 raft_consensus.cc:2454] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7853bae086514f24b6f02af165e9d9ab in term 0.
I20260925 15:29:06.251693 18819 raft_consensus.cc:2454] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7853bae086514f24b6f02af165e9d9ab in term 0.
I20260925 15:29:06.251883 18887 leader_election.cc:302] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7853bae086514f24b6f02af165e9d9ab, fb2e1b3dea254e6f8ca21480686dbd29; no voters:
I20260925 15:29:06.252022 19158 raft_consensus.cc:2781] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:06.252081 19158 raft_consensus.cc:482] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:06.252115 19158 raft_consensus.cc:3037] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:06.252887 19158 raft_consensus.cc:504] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.253023 19158 leader_election.cc:288] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [CANDIDATE]: Term 1 election: Requested vote from peers 57874a9004194f888b68b896d7c8d838 (127.1.163.65:33005), fb2e1b3dea254e6f8ca21480686dbd29 (127.1.163.67:36549)
I20260925 15:29:06.253190 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e734ab45c4044a57b8b2057a0a9e35a6" candidate_uuid: "7853bae086514f24b6f02af165e9d9ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57874a9004194f888b68b896d7c8d838"
I20260925 15:29:06.253219 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e734ab45c4044a57b8b2057a0a9e35a6" candidate_uuid: "7853bae086514f24b6f02af165e9d9ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2e1b3dea254e6f8ca21480686dbd29"
I20260925 15:29:06.253273 18819 raft_consensus.cc:3037] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:06.253290 19087 raft_consensus.cc:3037] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:06.254158 18819 raft_consensus.cc:2454] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7853bae086514f24b6f02af165e9d9ab in term 1.
I20260925 15:29:06.254158 19087 raft_consensus.cc:2454] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7853bae086514f24b6f02af165e9d9ab in term 1.
I20260925 15:29:06.254338 18887 leader_election.cc:302] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7853bae086514f24b6f02af165e9d9ab, fb2e1b3dea254e6f8ca21480686dbd29; no voters:
I20260925 15:29:06.254478 19158 raft_consensus.cc:2781] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:06.254628 19158 raft_consensus.cc:686] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 1 LEADER]: Becoming Leader. State: Replica: 7853bae086514f24b6f02af165e9d9ab, State: Running, Role: LEADER
I20260925 15:29:06.254729 19158 consensus_queue.cc:237] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } }
I20260925 15:29:06.255662 18672 catalog_manager.cc:5495] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 7853bae086514f24b6f02af165e9d9ab (127.1.163.66). New cstate: current_term: 1 leader_uuid: "7853bae086514f24b6f02af165e9d9ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7853bae086514f24b6f02af165e9d9ab" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 } health_report { overall_health: UNKNOWN } } }
I20260925 15:29:06.287956 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.285461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39672 (local address 127.1.163.66:39031)
0925 15:29:06.285527 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.285529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.285536 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:29:06.285573 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.285574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.285591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.285661 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:06.285864 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.286344 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.287132 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.287319 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.287376 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.287570 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:06.287674 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:06.287739 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.287851 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.287869 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:29:06.290060 18819 raft_consensus.cc:1261] T e734ab45c4044a57b8b2057a0a9e35a6 P 57874a9004194f888b68b896d7c8d838 [term 1 FOLLOWER]: Refusing update from remote peer 7853bae086514f24b6f02af165e9d9ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:06.290059 19087 raft_consensus.cc:1261] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 1 FOLLOWER]: Refusing update from remote peer 7853bae086514f24b6f02af165e9d9ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:06.290501 19158 consensus_queue.cc:1035] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2e1b3dea254e6f8ca21480686dbd29" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:06.290714 19160 consensus_queue.cc:1035] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "57874a9004194f888b68b896d7c8d838" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:06.293351 19163 mvcc.cc:205] Tried to move back new op lower bound from 7333274199200280576 to 7333274199059771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333274199200280576}]
I20260925 15:29:06.296700 18671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:06.296810 18671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:06.297734 18671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccd94e6eccfc4e38b1c33900b7d96457: Sending DeleteTablet for 3 replicas of tablet e734ab45c4044a57b8b2057a0a9e35a6
I20260925 15:29:06.298067 18933 tablet_service.cc:1514] Processing DeleteTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:06 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:39656
I20260925 15:29:06.298206 18799 tablet_service.cc:1514] Processing DeleteTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:06 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:39810
I20260925 15:29:06.298424 19067 tablet_service.cc:1514] Processing DeleteTablet for tablet e734ab45c4044a57b8b2057a0a9e35a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:06 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:40736
I20260925 15:29:06.298476 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18736
I20260925 15:29:06.298611 19172 tablet_replica.cc:330] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: stopping tablet replica
I20260925 15:29:06.298748 19172 raft_consensus.cc:2229] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:06.299011 19172 raft_consensus.cc:2258] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:06.300159 19172 ts_tablet_manager.cc:1905] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:06.302345 19172 ts_tablet_manager.cc:1918] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:06.302438 19172 log.cc:1196] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/e734ab45c4044a57b8b2057a0a9e35a6
I20260925 15:29:06.302723 19172 ts_tablet_manager.cc:1939] T e734ab45c4044a57b8b2057a0a9e35a6 P 7853bae086514f24b6f02af165e9d9ab: Deleting consensus metadata
I20260925 15:29:06.303200 18661 catalog_manager.cc:4841] TS 7853bae086514f24b6f02af165e9d9ab (127.1.163.66:39031): tablet e734ab45c4044a57b8b2057a0a9e35a6 (table test-table [id=46c1a5b7d12f462f8c9ce9e39ca78256]) successfully deleted
W20260925 15:29:06.304289 18659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:33005 (error 108)
I20260925 15:29:06.304486 19174 tablet_replica.cc:330] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: stopping tablet replica
I20260925 15:29:06.304662 19174 raft_consensus.cc:2229] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:06.304821 19174 raft_consensus.cc:2258] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:06.305034 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18870
I20260925 15:29:06.305666 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.305063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33005 (local address 127.0.0.1:39824)
0925 15:29:06.305554 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.305606 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464}
I20260925 15:29:06.306075 19174 ts_tablet_manager.cc:1905] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:29:06.307593 18659 catalog_manager.cc:4584] TS 57874a9004194f888b68b896d7c8d838 (127.1.163.65:33005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e734ab45c4044a57b8b2057a0a9e35a6: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33005: connect: Connection refused (error 111)
I20260925 15:29:06.308233 19174 ts_tablet_manager.cc:1918] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:06.308303 19174 log.cc:1196] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/e734ab45c4044a57b8b2057a0a9e35a6
I20260925 15:29:06.308681 19174 ts_tablet_manager.cc:1939] T e734ab45c4044a57b8b2057a0a9e35a6 P fb2e1b3dea254e6f8ca21480686dbd29: Deleting consensus metadata
I20260925 15:29:06.309135 18659 catalog_manager.cc:4841] TS fb2e1b3dea254e6f8ca21480686dbd29 (127.1.163.67:36549): tablet e734ab45c4044a57b8b2057a0a9e35a6 (table test-table [id=46c1a5b7d12f462f8c9ce9e39ca78256]) successfully deleted
I20260925 15:29:06.311583 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19004
I20260925 15:29:06.317159 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 18642
2026-09-25T15:29:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:06Z Disabled control of system clock
I20260925 15:29:06.346439 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37315
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:06.430573 19179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:06.430955 19179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:06.431001 19179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:06.432482 19179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:06.432598 19179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:06.432623 19179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:06.432637 19179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:06.432657 19179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:06.434388 19179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:06.434633 19179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:06.434928 19179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:06.440320 19187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.440290 19184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.440275 19185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.440721 19179 server_base.cc:1034] running on GCE node
I20260925 15:29:06.440996 19179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:06.441351 19179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:06.442523 19179 hybrid_clock.cc:648] HybridClock initialized: now 1790350146442482 us; error 49 us; skew 500 ppm
I20260925 15:29:06.443934 19179 webserver.cc:460] Webserver started at http://127.1.163.126:43419/ using document root <none> and password file <none>
I20260925 15:29:06.444219 19179 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:06.444321 19179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:06.444480 19179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:06.445663 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "25e064c1e95e43ceaec82e2c1abe6e67"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "8386942cf07c073418e4435192ee1a58"
server_key_iv: "af1824b708b33891f5d45afdf993dd64"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.446046 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "25e064c1e95e43ceaec82e2c1abe6e67"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "8386942cf07c073418e4435192ee1a58"
server_key_iv: "af1824b708b33891f5d45afdf993dd64"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.447539 19179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:06.448451 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.448746 19179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:06.448881 19179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "25e064c1e95e43ceaec82e2c1abe6e67"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "8386942cf07c073418e4435192ee1a58"
server_key_iv: "af1824b708b33891f5d45afdf993dd64"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.448976 19179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:06.473738 19179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:06.474123 19179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:06.474305 19179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:06.479067 19179 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37315
I20260925 15:29:06.479105 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37315 every 8 connection(s)
I20260925 15:29:06.479502 19179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:06.480720 19246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.481489 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19179
I20260925 15:29:06.481627 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:06.481853 1677 external_mini_cluster.cc:1428] Setting key a9acbe06da562d1e32ce697bb8c43072
I20260925 15:29:06.483768 19246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: Bootstrap starting.
I20260925 15:29:06.484521 19246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.484839 19246 log.cc:824] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.485628 19246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: No bootstrap required, opened a new log
I20260925 15:29:06.488130 19246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } }
I20260925 15:29:06.488315 19246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:06.488351 19246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e064c1e95e43ceaec82e2c1abe6e67, State: Initialized, Role: FOLLOWER
I20260925 15:29:06.488528 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.482491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.1.163.126:37315)
0925 15:29:06.482725 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.482734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.482766 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:29:06.483468 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.483478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.483561 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.483716 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.483962 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.485331 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.486203 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.486844 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.486988 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.487779 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.487785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.487786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:06.488014 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.488061 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:06.488077 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.488148 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.488209 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.488290 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260925 15:29:06.488504 19246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } }
I20260925 15:29:06.488611 19246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:06.488647 19246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:06.488700 19246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:06.489404 19246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } }
I20260925 15:29:06.489492 19246 leader_election.cc:302] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e064c1e95e43ceaec82e2c1abe6e67; no voters:
I20260925 15:29:06.489629 19246 leader_election.cc:288] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:06.489727 19251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:06.489945 19251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [term 1 LEADER]: Becoming Leader. State: Replica: 25e064c1e95e43ceaec82e2c1abe6e67, State: Running, Role: LEADER
I20260925 15:29:06.489959 19246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:06.490056 19251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } }
I20260925 15:29:06.490453 19253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e064c1e95e43ceaec82e2c1abe6e67. Latest consensus state: current_term: 1 leader_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } } }
I20260925 15:29:06.490554 19253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:06.490710 19252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e064c1e95e43ceaec82e2c1abe6e67" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37315 } } }
I20260925 15:29:06.490782 19252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:06.490965 19260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:06.491457 19260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:06.493251 19260 catalog_manager.cc:1347] Generated new cluster ID: 53f98b92fe23453da9996a81a80587b5
I20260925 15:29:06.493315 19260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:06.544734 19260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:06.545336 19260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:06.547920 19260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: Generated new TSK 0
I20260925 15:29:06.548120 19260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:06.551515 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--builtin_ntp_servers=127.1.163.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:06.641618 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:06.641851 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:06.641875 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:06.643067 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:06.643108 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:06.643136 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:06.644512 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:06.644836 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:06.645125 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:06.650529 19275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.650525 19278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.650525 19276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.650763 19270 server_base.cc:1034] running on GCE node
I20260925 15:29:06.650997 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:06.651219 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:06.652360 19270 hybrid_clock.cc:648] HybridClock initialized: now 1790350146652333 us; error 46 us; skew 500 ppm
I20260925 15:29:06.654203 19270 webserver.cc:460] Webserver started at http://127.1.163.65:42451/ using document root <none> and password file <none>
I20260925 15:29:06.654538 19270 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:06.654609 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:06.654762 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:06.656256 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "f550171b5a524b4b9b3f7652569f66a0"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "47746fa672e132d9eb2eeaf2da29ae58"
server_key_iv: "90a576a0e92be842023e5d7098f2230b"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.656737 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "f550171b5a524b4b9b3f7652569f66a0"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "47746fa672e132d9eb2eeaf2da29ae58"
server_key_iv: "90a576a0e92be842023e5d7098f2230b"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.658154 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:06.658955 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.659164 19270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:06.659255 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "f550171b5a524b4b9b3f7652569f66a0"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "47746fa672e132d9eb2eeaf2da29ae58"
server_key_iv: "90a576a0e92be842023e5d7098f2230b"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.659309 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:06.677286 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:06.677577 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:06.677686 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:06.677881 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:06.678222 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:06.678251 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.678275 19270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:06.678287 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.683965 19270 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:38917
I20260925 15:29:06.684031 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:38917 every 8 connection(s)
I20260925 15:29:06.684366 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:29:06.687341 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19270
I20260925 15:29:06.687466 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:06.687664 1677 external_mini_cluster.cc:1428] Setting key 6d5e458c58cb18f3c104c0d8f0038472
I20260925 15:29:06.688660 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--builtin_ntp_servers=127.1.163.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:06.690800 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.685389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:56415 (local address 127.1.163.126:37315)
0925 15:29:06.685512 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.685514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.685681 (+ 167us) server_negotiation.cc:407] Connection header received
0925 15:29:06.686298 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.686300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.686341 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.686433 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.687246 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.687931 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.689077 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.689283 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.689375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.690369 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.690370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.690370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:06.690375 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.690391 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:06.690395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.690465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.690684 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.690719 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:29:06.691084 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.685319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37315 (local address 127.1.163.65:56415)
0925 15:29:06.685545 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.685582 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.686157 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.686551 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.686556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.686573 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:06.687129 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.687136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.688034 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.688036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.688975 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.688978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.689387 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.689388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.689533 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.690205 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:06.690205 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:06.690236 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:06.690262 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:06.690492 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:06.690498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:06.690543 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.690681 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.690799 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260925 15:29:06.691642 19398 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37315
I20260925 15:29:06.691757 19398 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:06.691977 19398 heartbeater.cc:511] Master 127.1.163.126:37315 requested a full tablet report, sending...
I20260925 15:29:06.692720 19210 ts_manager.cc:194] Registered new tserver with Master: f550171b5a524b4b9b3f7652569f66a0 (127.1.163.65:38917)
I20260925 15:29:06.693387 19210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.65:56415
W20260925 15:29:06.775010 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:06.775225 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:06.775249 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:06.776520 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:06.776631 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:06.776664 19401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:06.778132 19401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:06.778398 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:06.778734 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:06.784087 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.784049 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.784251 19401 server_base.cc:1034] running on GCE node
W20260925 15:29:06.784067 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.784740 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:06.785003 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:06.786343 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790350146786327 us; error 42 us; skew 500 ppm
I20260925 15:29:06.787945 19401 webserver.cc:460] Webserver started at http://127.1.163.66:41185/ using document root <none> and password file <none>
I20260925 15:29:06.788216 19401 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:06.788290 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:06.788440 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:06.789691 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "da890e6d664f48188d9dc64e0ce28789"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "786090141392fb7dc76f57ea2e8c7a70"
server_key_iv: "d6018f9ee2cf61afdb2495aa01df6d69"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.790073 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "da890e6d664f48188d9dc64e0ce28789"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "786090141392fb7dc76f57ea2e8c7a70"
server_key_iv: "d6018f9ee2cf61afdb2495aa01df6d69"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.791651 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:29:06.792594 19415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.792778 19401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:06.792899 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "da890e6d664f48188d9dc64e0ce28789"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "786090141392fb7dc76f57ea2e8c7a70"
server_key_iv: "d6018f9ee2cf61afdb2495aa01df6d69"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.792997 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:06.823055 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:06.823427 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:06.823587 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:06.823838 19401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:06.824227 19401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:06.824285 19401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.824325 19401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:06.824353 19401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.830143 19401 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:46655
I20260925 15:29:06.830193 19528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:46655 every 8 connection(s)
I20260925 15:29:06.830554 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:29:06.835103 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19401
I20260925 15:29:06.835244 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:06.835495 1677 external_mini_cluster.cc:1428] Setting key 524aba3e39b8d157ed457dc004a6505a
I20260925 15:29:06.836778 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--builtin_ntp_servers=127.1.163.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:06.836860 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.831289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37315 (local address 127.1.163.66:56879)
0925 15:29:06.831492 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.831534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.832099 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.832356 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.832359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.832368 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:06.833001 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.833007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.833734 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.833735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.834560 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.834562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.835029 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.835030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.835227 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.835996 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:06.835997 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:06.836037 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:06.836074 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:06.836301 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:06.836307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:06.836311 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.836438 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.836583 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260925 15:29:06.837411 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.831364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:56879 (local address 127.1.163.126:37315)
0925 15:29:06.831465 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.831467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.831658 (+ 191us) server_negotiation.cc:407] Connection header received
0925 15:29:06.832198 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.832200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.832211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.832280 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.833094 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.833641 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.834648 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.834908 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.834975 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.836198 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.836199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.836200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:06.836205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.836223 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:06.836227 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.836301 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.837328 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.837353 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:29:06.837859 19529 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37315
I20260925 15:29:06.837980 19529 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:06.838229 19529 heartbeater.cc:511] Master 127.1.163.126:37315 requested a full tablet report, sending...
I20260925 15:29:06.838658 19210 ts_manager.cc:194] Registered new tserver with Master: da890e6d664f48188d9dc64e0ce28789 (127.1.163.66:46655)
I20260925 15:29:06.839064 19210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.66:56879
W20260925 15:29:06.923995 19532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:06.924317 19532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:06.924363 19532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:06.925894 19532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:06.926000 19532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:06.926050 19532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:06.927692 19532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:06.928026 19532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:06.928395 19532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:06.934772 19540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.934856 19537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:06.934762 19538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:06.935026 19532 server_base.cc:1034] running on GCE node
I20260925 15:29:06.935386 19532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:06.935798 19532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:06.936964 19532 hybrid_clock.cc:648] HybridClock initialized: now 1790350146936929 us; error 50 us; skew 500 ppm
I20260925 15:29:06.938525 19532 webserver.cc:460] Webserver started at http://127.1.163.67:41451/ using document root <none> and password file <none>
I20260925 15:29:06.938798 19532 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:06.938875 19532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:06.939023 19532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:06.940174 19532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "6e66792bc4754e89946cfe3df70996bf"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "54fb7b7a33df3fef426d57b23071e8e3"
server_key_iv: "8e805b6058b5f57e094eec2eb00b8e73"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.940641 19532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "6e66792bc4754e89946cfe3df70996bf"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "54fb7b7a33df3fef426d57b23071e8e3"
server_key_iv: "8e805b6058b5f57e094eec2eb00b8e73"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.942294 19532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:06.943295 19546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.943527 19532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:06.943648 19532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "6e66792bc4754e89946cfe3df70996bf"
format_stamp: "Formatted at 2026-09-25 15:29:06 on dist-test-slave-hvv5"
server_key: "54fb7b7a33df3fef426d57b23071e8e3"
server_key_iv: "8e805b6058b5f57e094eec2eb00b8e73"
server_key_version: "encryptionkey@0"
I20260925 15:29:06.943737 19532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:06.953279 19532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:06.953625 19532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:06.953784 19532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:06.954038 19532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:06.954393 19532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:06.954452 19532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.954530 19532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:06.954560 19532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:06.960512 19532 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:39913
I20260925 15:29:06.960583 19659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:39913 every 8 connection(s)
I20260925 15:29:06.960950 19532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:29:06.962818 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19532
I20260925 15:29:06.962956 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:06.963150 1677 external_mini_cluster.cc:1428] Setting key 7ed1515019f515c568477d981a5bc2c9
I20260925 15:29:06.967407 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.961718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:45879 (local address 127.1.163.126:37315)
0925 15:29:06.961784 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.961786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.962119 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:29:06.962718 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.962720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.962734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.962880 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.963871 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.964391 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.965588 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.965925 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.966060 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.967107 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.967108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.967109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:06.967114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.967130 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:06.967134 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.967190 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.967283 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.967322 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:29:06.967633 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.961733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37315 (local address 127.1.163.67:45879)
0925 15:29:06.961949 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.961987 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.962498 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.962976 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.962998 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.963010 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:06.963780 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.963789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.964499 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.964500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.965480 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.965483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.966073 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.966074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.966265 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.966935 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:06.966936 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:06.966967 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:06.966994 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:06.967204 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:06.967211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:06.967214 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.967278 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.967392 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260925 15:29:06.968202 19660 heartbeater.cc:347] Connected to a master server at 127.1.163.126:37315
I20260925 15:29:06.968300 19660 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:06.968461 19660 heartbeater.cc:511] Master 127.1.163.126:37315 requested a full tablet report, sending...
I20260925 15:29:06.968890 19210 ts_manager.cc:194] Registered new tserver with Master: 6e66792bc4754e89946cfe3df70996bf (127.1.163.67:39913)
I20260925 15:29:06.969260 19210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.67:45879
I20260925 15:29:06.975108 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:06.979530 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.976636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.1.163.126:37315)
0925 15:29:06.976711 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.976713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.976804 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:29:06.976845 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.976845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.976855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.976912 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:06.977084 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.977604 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.978362 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.978575 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.978634 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.979272 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:06.979274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:06.979274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:06.979278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:06.979292 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:06.979296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:06.979346 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.979382 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.979423 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:29:06.981189 19210 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:06.982800 19210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:06.990087 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:39913 (local address 127.0.0.1:43488)
0925 15:29:06.985784 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.985791 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.985827 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.986387 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.986388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.986390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.986435 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.986436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987691 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.987693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.989001 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.989004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989799 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.989864 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.989868 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.989901 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.989951 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260925 15:29:06.990170 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.1.163.66:46655)
0925 15:29:06.985793 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.985799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.985813 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:29:06.985855 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.985857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.985875 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.985972 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.986297 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987509 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.988838 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989585 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989669 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.989771 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.990008 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.990062 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260925 15:29:06.990166 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.1.163.65:38917)
0925 15:29:06.985947 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.985951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.985962 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:06.986005 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.986007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.986022 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.986101 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.986337 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987126 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989019 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989771 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989855 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.989938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.990039 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.990084 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":98,"threads_started":1}
I20260925 15:29:06.990092 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:46655 (local address 127.0.0.1:50928)
0925 15:29:06.985363 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.985417 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.985492 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.986094 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.986097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.986100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.986203 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.986205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987628 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.987629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.988735 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.988738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989689 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.989789 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.989806 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.989882 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.989950 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":7,"thread_start_us":69,"threads_started":1}
I20260925 15:29:06.990746 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.1.163.67:39913)
0925 15:29:06.986246 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:06.986251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:06.986260 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:06.986297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:06.986299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:06.986314 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:06.986388 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:06.986497 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987558 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989074 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989624 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.990461 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.990580 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:06.990607 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 15:29:06.990659 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":115,"threads_started":1}
I20260925 15:29:06.990860 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:06.985136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:38917 (local address 127.0.0.1:56948)
0925 15:29:06.985513 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:06.985525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:06.985584 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:06.986115 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:06.986116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:06.986118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:06.986211 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.986213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.987205 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.987206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.988917 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:06.988920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:06.989864 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:06.989865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:06.989938 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:06.989942 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:06.990347 (+ 405us) client_negotiation.cc:240] Negotiation successful
0925 15:29:06.990505 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":23,"thread_start_us":74,"threads_started":1}
I20260925 15:29:06.991565 19594 tablet_service.cc:1467] Processing CreateTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a (DEFAULT_TABLE table=test-table [id=6af7cf7483564430aa0e32b4fa02e3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.991607 19332 tablet_service.cc:1467] Processing CreateTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a (DEFAULT_TABLE table=test-table [id=6af7cf7483564430aa0e32b4fa02e3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.991961 19332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70537a52dcdb4d17849a81ce831b6e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.991961 19594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70537a52dcdb4d17849a81ce831b6e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.993836 19463 tablet_service.cc:1467] Processing CreateTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a (DEFAULT_TABLE table=test-table [id=6af7cf7483564430aa0e32b4fa02e3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:06.994509 19463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70537a52dcdb4d17849a81ce831b6e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:06.995692 19674 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Bootstrap starting.
I20260925 15:29:06.996479 19675 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Bootstrap starting.
I20260925 15:29:06.996670 19674 tablet_bootstrap.cc:654] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.997087 19674 log.cc:824] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.997208 19675 tablet_bootstrap.cc:654] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.997498 19675 log.cc:824] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:06.998282 19675 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: No bootstrap required, opened a new log
I20260925 15:29:06.998396 19675 ts_tablet_manager.cc:1397] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:06.998445 19678 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Bootstrap starting.
I20260925 15:29:06.999050 19678 tablet_bootstrap.cc:654] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:06.999120 19674 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: No bootstrap required, opened a new log
I20260925 15:29:06.999213 19674 ts_tablet_manager.cc:1397] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:29:06.999415 19678 log.cc:824] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:07.000306 19678 tablet_bootstrap.cc:492] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: No bootstrap required, opened a new log
I20260925 15:29:07.000396 19678 ts_tablet_manager.cc:1397] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:07.001579 19675 raft_consensus.cc:348] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.001780 19675 raft_consensus.cc:374] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:07.001828 19675 raft_consensus.cc:729] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f550171b5a524b4b9b3f7652569f66a0, State: Initialized, Role: FOLLOWER
I20260925 15:29:07.001966 19675 consensus_queue.cc:260] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.002283 19675 ts_tablet_manager.cc:1428] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:29:07.002185 19674 raft_consensus.cc:348] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.002319 19398 heartbeater.cc:503] Master 127.1.163.126:37315 was elected leader, sending a full tablet report...
I20260925 15:29:07.002351 19674 raft_consensus.cc:374] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:07.002388 19674 raft_consensus.cc:729] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e66792bc4754e89946cfe3df70996bf, State: Initialized, Role: FOLLOWER
I20260925 15:29:07.002523 19674 consensus_queue.cc:260] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.002526 19678 raft_consensus.cc:348] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.002692 19678 raft_consensus.cc:374] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:07.002736 19678 raft_consensus.cc:729] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da890e6d664f48188d9dc64e0ce28789, State: Initialized, Role: FOLLOWER
I20260925 15:29:07.002774 19674 ts_tablet_manager.cc:1428] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260925 15:29:07.002876 19678 consensus_queue.cc:260] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.002966 19660 heartbeater.cc:503] Master 127.1.163.126:37315 was elected leader, sending a full tablet report...
I20260925 15:29:07.003139 19678 ts_tablet_manager.cc:1428] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:07.003294 19529 heartbeater.cc:503] Master 127.1.163.126:37315 was elected leader, sending a full tablet report...
W20260925 15:29:07.081118 19530 tablet.cc:2369] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:29:07.185173 19399 tablet.cc:2369] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:29:07.211529 19661 tablet.cc:2369] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:07.335023 19682 raft_consensus.cc:482] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:07.335160 19682 raft_consensus.cc:504] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.335638 19682 leader_election.cc:288] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f550171b5a524b4b9b3f7652569f66a0 (127.1.163.65:38917), 6e66792bc4754e89946cfe3df70996bf (127.1.163.67:39913)
I20260925 15:29:07.340183 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.336160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:58875 (local address 127.1.163.67:39913)
0925 15:29:07.336205 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:07.336207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:07.336341 (+ 134us) server_negotiation.cc:407] Connection header received
0925 15:29:07.336379 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:07.336380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:07.336391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:07.336455 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:07.336641 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.337425 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339110 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.339800 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:07.339881 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:07.340053 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:29:07.340103 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:29:07.340209 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.335949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:38917 (local address 127.1.163.66:52287)
0925 15:29:07.336047 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:07.336062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:07.336139 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:07.336405 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:07.336406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:07.336407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:07.336516 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:07.336518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.337558 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.337560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:07.338985 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:07.338989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.339825 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:07.339949 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:07.339954 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:29:07.340031 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:29:07.340094 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":92}
I20260925 15:29:07.340183 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.336175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:52287 (local address 127.1.163.65:38917)
0925 15:29:07.336237 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:07.336239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:07.336248 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:07.336289 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:07.336290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:07.336324 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:07.336404 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:07.336652 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.337440 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339110 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.339714 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:07.339816 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:07.340053 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:29:07.340105 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:29:07.340216 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.336062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:39913 (local address 127.1.163.66:58875)
0925 15:29:07.336254 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:07.336262 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:07.336316 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:07.336474 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:07.336475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:07.336476 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:07.336562 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:07.336564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.337558 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.337560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:07.338985 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:07.338989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.339910 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.339910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:07.339972 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:07.339976 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:07.340031 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:29:07.340094 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":89,"thread_start_us":70,"threads_started":1}
I20260925 15:29:07.340646 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70537a52dcdb4d17849a81ce831b6e3a" candidate_uuid: "da890e6d664f48188d9dc64e0ce28789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f550171b5a524b4b9b3f7652569f66a0" is_pre_election: true
I20260925 15:29:07.340802 19352 raft_consensus.cc:2454] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da890e6d664f48188d9dc64e0ce28789 in term 0.
I20260925 15:29:07.341064 19417 leader_election.cc:302] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da890e6d664f48188d9dc64e0ce28789, f550171b5a524b4b9b3f7652569f66a0; no voters:
I20260925 15:29:07.341208 19682 raft_consensus.cc:2781] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:07.341259 19682 raft_consensus.cc:482] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:07.341286 19682 raft_consensus.cc:3037] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:07.341220 19614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70537a52dcdb4d17849a81ce831b6e3a" candidate_uuid: "da890e6d664f48188d9dc64e0ce28789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e66792bc4754e89946cfe3df70996bf" is_pre_election: true
I20260925 15:29:07.341387 19614 raft_consensus.cc:2454] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da890e6d664f48188d9dc64e0ce28789 in term 0.
I20260925 15:29:07.342139 19682 raft_consensus.cc:504] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.342283 19682 leader_election.cc:288] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [CANDIDATE]: Term 1 election: Requested vote from peers f550171b5a524b4b9b3f7652569f66a0 (127.1.163.65:38917), 6e66792bc4754e89946cfe3df70996bf (127.1.163.67:39913)
I20260925 15:29:07.342427 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70537a52dcdb4d17849a81ce831b6e3a" candidate_uuid: "da890e6d664f48188d9dc64e0ce28789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f550171b5a524b4b9b3f7652569f66a0"
I20260925 15:29:07.342500 19614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70537a52dcdb4d17849a81ce831b6e3a" candidate_uuid: "da890e6d664f48188d9dc64e0ce28789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e66792bc4754e89946cfe3df70996bf"
I20260925 15:29:07.342537 19352 raft_consensus.cc:3037] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:07.342584 19614 raft_consensus.cc:3037] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:07.343461 19614 raft_consensus.cc:2454] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da890e6d664f48188d9dc64e0ce28789 in term 1.
I20260925 15:29:07.343461 19352 raft_consensus.cc:2454] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da890e6d664f48188d9dc64e0ce28789 in term 1.
I20260925 15:29:07.343676 19417 leader_election.cc:302] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e66792bc4754e89946cfe3df70996bf, da890e6d664f48188d9dc64e0ce28789; no voters:
I20260925 15:29:07.343778 19682 raft_consensus.cc:2781] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:07.343930 19682 raft_consensus.cc:686] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 1 LEADER]: Becoming Leader. State: Replica: da890e6d664f48188d9dc64e0ce28789, State: Running, Role: LEADER
I20260925 15:29:07.344045 19682 consensus_queue.cc:237] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } }
I20260925 15:29:07.345005 19209 catalog_manager.cc:5495] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 reported cstate change: term changed from 0 to 1, leader changed from <none> to da890e6d664f48188d9dc64e0ce28789 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "da890e6d664f48188d9dc64e0ce28789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da890e6d664f48188d9dc64e0ce28789" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46655 } health_report { overall_health: HEALTHY } } }
I20260925 15:29:07.436757 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.434105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.1.163.66:46655)
0925 15:29:07.434199 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:07.434201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:07.434211 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:07.434250 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:07.434251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:07.434272 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:07.434357 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:07.434520 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.435035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.435790 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:07.436036 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:07.436099 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:07.436238 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:07.436346 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:07.436424 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:07.436612 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:29:07.436633 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:29:07.438766 19352 raft_consensus.cc:1261] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 1 FOLLOWER]: Refusing update from remote peer da890e6d664f48188d9dc64e0ce28789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:07.438755 19614 raft_consensus.cc:1261] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 1 FOLLOWER]: Refusing update from remote peer da890e6d664f48188d9dc64e0ce28789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:07.439160 19682 consensus_queue.cc:1035] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f550171b5a524b4b9b3f7652569f66a0" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 38917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:07.439329 19684 consensus_queue.cc:1035] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e66792bc4754e89946cfe3df70996bf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:07.442082 19688 mvcc.cc:205] Tried to move back new op lower bound from 7333274203905892352 to 7333274203521761280. Current Snapshot: MvccSnapshot[applied={T|T < 7333274203905892352}]
I20260925 15:29:07.444645 19210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:07.444784 19210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:07.445545 19210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25e064c1e95e43ceaec82e2c1abe6e67: Sending DeleteTablet for 3 replicas of tablet 70537a52dcdb4d17849a81ce831b6e3a
I20260925 15:29:07.445875 19332 tablet_service.cc:1514] Processing DeleteTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:07 UTC) from {username='slave'} at 127.0.0.1:56948
I20260925 15:29:07.445930 19463 tablet_service.cc:1514] Processing DeleteTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:07 UTC) from {username='slave'} at 127.0.0.1:50928
I20260925 15:29:07.445926 19594 tablet_service.cc:1514] Processing DeleteTablet for tablet 70537a52dcdb4d17849a81ce831b6e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:07 UTC) from {username='slave'} at 127.0.0.1:43488
I20260925 15:29:07.446128 19698 tablet_replica.cc:330] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0: stopping tablet replica
I20260925 15:29:07.446235 19698 raft_consensus.cc:2229] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:07.446374 19698 raft_consensus.cc:2258] T 70537a52dcdb4d17849a81ce831b6e3a P f550171b5a524b4b9b3f7652569f66a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:07.446398 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19270
W20260925 15:29:07.451815 19195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:38917 (error 108)
I20260925 15:29:07.451879 19697 tablet_replica.cc:330] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789: stopping tablet replica
I20260925 15:29:07.451992 19697 raft_consensus.cc:2229] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:07.452163 19699 tablet_replica.cc:330] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: stopping tablet replica
I20260925 15:29:07.452279 19699 raft_consensus.cc:2229] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:07.452396 19699 raft_consensus.cc:2258] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:07.452423 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19401
I20260925 15:29:07.452631 19697 raft_consensus.cc:2258] T 70537a52dcdb4d17849a81ce831b6e3a P da890e6d664f48188d9dc64e0ce28789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:07.454263 19699 ts_tablet_manager.cc:1905] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:07.456718 19699 ts_tablet_manager.cc:1918] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:07.456837 19699 log.cc:1196] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/70537a52dcdb4d17849a81ce831b6e3a
I20260925 15:29:07.457141 19699 ts_tablet_manager.cc:1939] T 70537a52dcdb4d17849a81ce831b6e3a P 6e66792bc4754e89946cfe3df70996bf: Deleting consensus metadata
I20260925 15:29:07.457237 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.457069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:38917 (local address 127.0.0.1:56958)
0925 15:29:07.457134 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:07.457187 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:29:07.457392 19195 catalog_manager.cc:4584] TS f550171b5a524b4b9b3f7652569f66a0 (127.1.163.65:38917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70537a52dcdb4d17849a81ce831b6e3a: Network error: Client connection negotiation failed: client connection to 127.1.163.65:38917: connect: Connection refused (error 111)
I20260925 15:29:07.457792 19195 catalog_manager.cc:4841] TS 6e66792bc4754e89946cfe3df70996bf (127.1.163.67:39913): tablet 70537a52dcdb4d17849a81ce831b6e3a (table test-table [id=6af7cf7483564430aa0e32b4fa02e3e9]) successfully deleted
I20260925 15:29:07.458669 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:07.458540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:46655 (local address 127.0.0.1:50952)
0925 15:29:07.458601 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:07.458633 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.66:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:29:07.458680 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19532
I20260925 15:29:07.463658 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19179
2026-09-25T15:29:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1145 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:07 dist-test-slave-hvv5 krb5kdc[19703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:07 dist-test-slave-hvv5 krb5kdc[19703](info): set up 2 sockets
Sep 25 15:29:07 dist-test-slave-hvv5 krb5kdc[19703](info): commencing operation
krb5kdc: starting...
W20260925 15:29:08.516353 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:08 dist-test-slave-hvv5 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:08.648968 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34965
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:34965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:08.737682 19716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:08.737967 19716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:08.738009 19716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:08.739256 19716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:08.739336 19716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:08.739362 19716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:08.739393 19716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:08.739413 19716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:08.741070 19716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:34965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:08.741389 19716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:08.741645 19716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:08.747052 19721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:08.747087 19724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:08.747071 19722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:08.747502 19716 server_base.cc:1034] running on GCE node
I20260925 15:29:08.747777 19716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:08.748121 19716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:08.749328 19716 hybrid_clock.cc:648] HybridClock initialized: now 1790350148749312 us; error 47 us; skew 500 ppm
Sep 25 15:29:08 dist-test-slave-hvv5 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:08.758455 19716 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:08.758994 19716 webserver.cc:460] Webserver started at http://127.1.163.126:33321/ using document root <none> and password file <none>
I20260925 15:29:08.759256 19716 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:08.759328 19716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:08.759482 19716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:08.760667 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "75986b8a9d2d41da887925b2b0d7d10e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "f1f866de6d95c8706da91343e756bd9b"
server_key_iv: "2b82c64d111743b7e3ef8e03b919d91f"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.761052 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "75986b8a9d2d41da887925b2b0d7d10e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "f1f866de6d95c8706da91343e756bd9b"
server_key_iv: "2b82c64d111743b7e3ef8e03b919d91f"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.762447 19716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:29:08.763329 19731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:08.763571 19716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:08.763721 19716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "75986b8a9d2d41da887925b2b0d7d10e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "f1f866de6d95c8706da91343e756bd9b"
server_key_iv: "2b82c64d111743b7e3ef8e03b919d91f"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.763829 19716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:08.777375 19716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:08.779770 19716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:08.779987 19716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:08.784989 19716 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:34965
I20260925 15:29:08.785058 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:34965 every 8 connection(s)
I20260925 15:29:08.785458 19716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:08.786369 19784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:08.788818 19784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: Bootstrap starting.
I20260925 15:29:08.789439 19784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:08.789742 19784 log.cc:824] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:08.790598 19784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: No bootstrap required, opened a new log
I20260925 15:29:08.792418 19784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } }
I20260925 15:29:08.792670 19784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:08.792701 19784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75986b8a9d2d41da887925b2b0d7d10e, State: Initialized, Role: FOLLOWER
I20260925 15:29:08.792893 19784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } }
I20260925 15:29:08.792966 19784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:08.792990 19784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:08.793066 19784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:08.793735 19784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } }
I20260925 15:29:08.793860 19784 leader_election.cc:302] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75986b8a9d2d41da887925b2b0d7d10e; no voters:
I20260925 15:29:08.794061 19784 leader_election.cc:288] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:08.794085 19787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:08.794317 19787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [term 1 LEADER]: Becoming Leader. State: Replica: 75986b8a9d2d41da887925b2b0d7d10e, State: Running, Role: LEADER
I20260925 15:29:08.794332 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19716
I20260925 15:29:08.794359 19784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:08.794450 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:08.794458 19787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } }
I20260925 15:29:08.794655 1677 external_mini_cluster.cc:1428] Setting key dbd24cf447bfe25a47833969cd7c97b1
I20260925 15:29:08.794838 19787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75986b8a9d2d41da887925b2b0d7d10e. Latest consensus state: current_term: 1 leader_uuid: "75986b8a9d2d41da887925b2b0d7d10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } } }
I20260925 15:29:08.794906 19787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:08.795044 19788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75986b8a9d2d41da887925b2b0d7d10e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75986b8a9d2d41da887925b2b0d7d10e" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34965 } } }
I20260925 15:29:08.795137 19788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:08.795616 19793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:08.796271 19793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:08.799305 19793 catalog_manager.cc:1347] Generated new cluster ID: e3bb2123da424fffb66c61aead8b3681
I20260925 15:29:08.799386 19793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:29:08 dist-test-slave-hvv5 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:08.806265 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:08.795324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.1.163.126:34965)
0925 15:29:08.796141 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:08.796153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:08.796194 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:29:08.796335 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:08.796343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:08.796421 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:08.796610 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:08.797219 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:08.798752 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:08.799686 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:08.800145 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:08.800278 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:08.802709 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:08.802713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:08.802713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:08.802727 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:08.804503 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:08.804954 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:08.804958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:08.804958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:08.805068 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:08.805287 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:08.805288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:08.805288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:08.805548 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:08.805716 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:08.805821 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:29:08.806089 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":110,"threads_started":1}
I20260925 15:29:08.813206 19793 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:08.813879 19793 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:08.823236 19793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: Generated new TSK 0
I20260925 15:29:08.823576 19793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:29:08.860035 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:08.963105 19810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:08.963335 19810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:08.963362 19810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:08.964505 19810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:08.964564 19810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:08.964593 19810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:08.966152 19810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:08.966410 19810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:08.966686 19810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:08.972316 19818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:08.972316 19816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:08.972486 19810 server_base.cc:1034] running on GCE node
W20260925 15:29:08.972759 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:08.973012 19810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:08.973316 19810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:08.974493 19810 hybrid_clock.cc:648] HybridClock initialized: now 1790350148974478 us; error 32 us; skew 500 ppm
Sep 25 15:29:08 dist-test-slave-hvv5 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:08.977221 19810 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:29:08.977824 19810 webserver.cc:460] Webserver started at http://127.1.163.65:38027/ using document root <none> and password file <none>
I20260925 15:29:08.978088 19810 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:08.978163 19810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:08.978324 19810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:08.979419 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "8e3d0bdc35594ce599fbd9338f71a00e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "b9c51f3dd313e20d3ab35a15f13b7e39"
server_key_iv: "6616b38b62fab81ce70312e61381be90"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.979805 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "8e3d0bdc35594ce599fbd9338f71a00e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "b9c51f3dd313e20d3ab35a15f13b7e39"
server_key_iv: "6616b38b62fab81ce70312e61381be90"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.981302 19810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:08.982079 19825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:08.982254 19810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:29:08.982354 19810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "8e3d0bdc35594ce599fbd9338f71a00e"
format_stamp: "Formatted at 2026-09-25 15:29:08 on dist-test-slave-hvv5"
server_key: "b9c51f3dd313e20d3ab35a15f13b7e39"
server_key_iv: "6616b38b62fab81ce70312e61381be90"
server_key_version: "encryptionkey@0"
I20260925 15:29:08.982447 19810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.007033 19810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.009563 19810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.009770 19810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.010088 19810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:09.010442 19810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:09.010499 19810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.010541 19810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:09.010571 19810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.017025 19810 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:33035
I20260925 15:29:09.017071 19938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:33035 every 8 connection(s)
I20260925 15:29:09.017433 19810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:29:09 dist-test-slave-hvv5 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350148, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:09.026965 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19810
I20260925 15:29:09.027086 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:09.027324 1677 external_mini_cluster.cc:1428] Setting key 93ef3517f939c8271099703fdb115413
I20260925 15:29:09.027670 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.018221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:39287 (local address 127.1.163.126:34965)
0925 15:29:09.018304 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.018306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.018530 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:29:09.019086 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.019088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.019101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.019195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.020248 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.020924 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.021919 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.022112 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.022167 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.024532 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.024570 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.024570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:09.024579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.025998 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.026437 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.026438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.026438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.026495 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.026793 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.026794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.026794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.026984 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.027065 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.027462 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.027569 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:29:09.027918 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.018201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34965 (local address 127.1.163.65:39287)
0925 15:29:09.018391 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.018436 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.018936 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.019255 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.019259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.019899 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:09.020108 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.020113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.021017 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.021018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.021834 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.021837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.022190 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.022191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.022293 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.022852 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:09.022861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:09.024439 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:09.026085 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.026093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.026094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.026350 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.026581 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.026582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.026582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.026704 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.027092 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:09.027100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:09.027289 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.027419 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.027641 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260925 15:29:09.028481 19939 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34965
I20260925 15:29:09.028617 19939 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:09.028827 19939 heartbeater.cc:511] Master 127.1.163.126:34965 requested a full tablet report, sending...
I20260925 15:29:09.029429 19748 ts_manager.cc:194] Registered new tserver with Master: 8e3d0bdc35594ce599fbd9338f71a00e (127.1.163.65:33035)
I20260925 15:29:09.030119 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:39287
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:29:09.056017 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:09.138717 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:09.138940 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:09.138965 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:09.140087 19944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:09.140125 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:09.140149 19944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:09.141567 19944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:09.141758 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:09.141934 19944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:09.147163 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.147173 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.147173 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.147437 19944 server_base.cc:1034] running on GCE node
I20260925 15:29:09.147655 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:09.147910 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:09.149098 19944 hybrid_clock.cc:648] HybridClock initialized: now 1790350149149086 us; error 30 us; skew 500 ppm
Sep 25 15:29:09 dist-test-slave-hvv5 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:09.151885 19944 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:29:09.152341 19944 webserver.cc:460] Webserver started at http://127.1.163.66:40459/ using document root <none> and password file <none>
I20260925 15:29:09.152627 19944 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:09.152697 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:09.152855 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:09.153973 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "38819747e02f4d478deac6a4708bbdf6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "e7c07e279ee528db71912fe0289ee391"
server_key_iv: "5c17fdb56226768f5c573d2355b10a95"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.154338 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "38819747e02f4d478deac6a4708bbdf6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "e7c07e279ee528db71912fe0289ee391"
server_key_iv: "5c17fdb56226768f5c573d2355b10a95"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.155704 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:09.156441 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.156646 19944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:29:09.156749 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "38819747e02f4d478deac6a4708bbdf6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "e7c07e279ee528db71912fe0289ee391"
server_key_iv: "5c17fdb56226768f5c573d2355b10a95"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.156838 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.167790 19944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.170439 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.170594 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.170847 19944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:09.171177 19944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:09.171211 19944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.171257 19944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:09.171281 19944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.177376 19944 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:46853
I20260925 15:29:09.177459 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:46853 every 8 connection(s)
I20260925 15:29:09.177858 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:29:09.181428 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 19944
I20260925 15:29:09.181546 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:09.181756 1677 external_mini_cluster.cc:1428] Setting key cdea540db4cf02f15bbb05ca02b4c9bb
Sep 25 15:29:09 dist-test-slave-hvv5 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:09.189380 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.178676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:59221 (local address 127.1.163.126:34965)
0925 15:29:09.178720 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.178722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.178968 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:29:09.179426 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.179427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.179437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.179500 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.180502 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.181161 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.182555 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.182827 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.182903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.185568 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.185570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.185570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:09.185579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.187531 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.187957 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.187964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.187965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.188066 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.188421 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.188423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.188423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.188650 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.188738 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.189126 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.189313 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:29:09.189568 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.178632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34965 (local address 127.1.163.66:59221)
0925 15:29:09.178791 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.178833 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.179338 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.179560 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.179564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.180179 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:09.180376 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.180382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.181272 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.181273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.182465 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.182468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.182924 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.182925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.183065 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.183670 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:09.183679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:09.185461 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:09.187629 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.187637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.187639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.187878 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.188163 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.188164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.188165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.188325 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.188750 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:09.188758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:09.188961 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.189090 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.189285 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 15:29:09.190099 20073 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34965
I20260925 15:29:09.190200 20073 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:09.190418 20073 heartbeater.cc:511] Master 127.1.163.126:34965 requested a full tablet report, sending...
I20260925 15:29:09.190860 19748 ts_manager.cc:194] Registered new tserver with Master: 38819747e02f4d478deac6a4708bbdf6 (127.1.163.66:46853)
I20260925 15:29:09.191430 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:59221
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:29:09.212288 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:09.295639 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:09.295862 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:09.295887 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:09.297254 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:09.297312 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:09.297341 20078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:09.298908 20078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:09.299132 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:09.299382 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:09.305158 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.305186 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.305297 20078 server_base.cc:1034] running on GCE node
W20260925 15:29:09.305459 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.305719 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:09.305987 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:09.307176 20078 hybrid_clock.cc:648] HybridClock initialized: now 1790350149307151 us; error 45 us; skew 500 ppm
Sep 25 15:29:09 dist-test-slave-hvv5 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:09.310197 20078 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:29:09.310721 20078 webserver.cc:460] Webserver started at http://127.1.163.67:42429/ using document root <none> and password file <none>
I20260925 15:29:09.310999 20078 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:09.311072 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:09.311223 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:09.312336 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "cecb9a5889824db0a81b483aa449df3f"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "0d79b573448269ca6965e1dfad39c7eb"
server_key_iv: "552ea1bb4159ee2054a7e48970842e8d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.312767 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "cecb9a5889824db0a81b483aa449df3f"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "0d79b573448269ca6965e1dfad39c7eb"
server_key_iv: "552ea1bb4159ee2054a7e48970842e8d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.314188 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:09.315068 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.315335 20078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:09.315455 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "cecb9a5889824db0a81b483aa449df3f"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "0d79b573448269ca6965e1dfad39c7eb"
server_key_iv: "552ea1bb4159ee2054a7e48970842e8d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.315538 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.323724 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.326328 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.326512 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.326752 20078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:09.327080 20078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:09.327135 20078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.327178 20078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:09.327209 20078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.333122 20078 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:40771
I20260925 15:29:09.333192 20206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:40771 every 8 connection(s)
I20260925 15:29:09.333528 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:29:09.337466 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20078
I20260925 15:29:09.337602 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:09.337819 1677 external_mini_cluster.cc:1428] Setting key 27539f596ea843e0434fcbf58713edc1
Sep 25 15:29:09 dist-test-slave-hvv5 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:09.343055 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.334294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:53765 (local address 127.1.163.126:34965)
0925 15:29:09.334401 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.334403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.334603 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:29:09.335126 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.335127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.335137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.335205 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.336139 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.336740 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.337798 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.338054 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.338143 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.340572 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.340574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.340575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:09.340583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.341827 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.342193 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.342194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.342195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.342245 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.342482 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.342483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.342484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.342614 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.342680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.342915 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.342987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:29:09.343274 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.334282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34965 (local address 127.1.163.67:53765)
0925 15:29:09.334464 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.334509 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.335041 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.335262 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.335265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.335846 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:09.336040 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.336044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.336834 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.336835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.337696 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.337699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.338165 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.338166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.338373 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.339030 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:09.339039 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:09.340451 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:09.341910 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.341916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.341918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.342129 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.342302 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:09.342302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:09.342302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:09.342401 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:09.342680 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:09.342684 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:09.342801 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.342885 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.343013 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":109,"threads_started":1}
I20260925 15:29:09.343779 20207 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34965
I20260925 15:29:09.343871 20207 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:09.344015 20207 heartbeater.cc:511] Master 127.1.163.126:34965 requested a full tablet report, sending...
I20260925 15:29:09.344369 19748 ts_manager.cc:194] Registered new tserver with Master: cecb9a5889824db0a81b483aa449df3f (127.1.163.67:40771)
I20260925 15:29:09.344772 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:53765
I20260925 15:29:09.349690 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:09.357615 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.351123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.1.163.126:34965)
0925 15:29:09.351185 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.351187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.351346 (+ 159us) server_negotiation.cc:407] Connection header received
0925 15:29:09.351387 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.351387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.351397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.351446 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.351992 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.352475 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.353254 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.353447 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.353501 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.354877 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.354878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.354878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:09.354883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.356406 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.356909 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.356909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.356910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.356962 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:09.357154 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:09.357155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:09.357155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:09.357290 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.357380 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.357495 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.357548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:29:09.359515 19748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:09.361474 19748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:09.368868 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:46853 (local address 127.0.0.1:42032)
0925 15:29:09.364666 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.364677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.364718 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.365064 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.365065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.365067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:09.365112 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.365113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366455 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.366456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.367473 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.367476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.368526 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.368527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.368629 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.368645 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.368708 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.368806 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260925 15:29:09.368876 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.1.163.66:46853)
0925 15:29:09.364896 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.364899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.364908 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:09.364946 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.364948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.364964 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.365180 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:09.365246 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366304 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.367613 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.368430 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.368510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.368597 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.368742 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.368785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260925 15:29:09.369335 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:40771 (local address 127.0.0.1:47044)
0925 15:29:09.364887 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.364898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.364956 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.365306 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.365306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.365308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:09.365353 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.365359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366246 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.366255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.367636 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.367639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.369116 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.369117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.369175 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.369178 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.369222 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.369262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":45,"thread_start_us":114,"threads_started":1}
I20260925 15:29:09.369577 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.1.163.67:40771)
0925 15:29:09.364729 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.364734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.365067 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:29:09.365113 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.365116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.365132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.365204 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:09.365446 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366129 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.367733 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.368430 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.368510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.368597 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.369456 (+ 859us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.369506 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
I20260925 15:29:09.369768 20007 tablet_service.cc:1467] Processing CreateTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd (DEFAULT_TABLE table=test-table [id=daec5c03943f478abc5cb909b208aed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:09.369917 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33035 (local address 127.0.0.1:48870)
0925 15:29:09.364414 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.364456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.364525 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.364874 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.364877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.364879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:09.364971 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.364972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366347 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.366348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.367637 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.367639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.369612 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.369613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.369731 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.369734 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.369809 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.369853 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":51,"thread_start_us":80,"threads_started":1}
I20260925 15:29:09.369925 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.364484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.1.163.65:33035)
0925 15:29:09.364667 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.364672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.364683 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:09.364726 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.364728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.364746 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.364836 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:09.365067 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.366239 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.368644 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.369453 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.369524 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.369576 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.369826 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.369864 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260925 15:29:09.370159 20007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbcd45a86a74e1da9c22f114be1cdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:09.372843 20224 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Bootstrap starting.
I20260925 15:29:09.373556 20224 tablet_bootstrap.cc:654] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:09.373502 19873 tablet_service.cc:1467] Processing CreateTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd (DEFAULT_TABLE table=test-table [id=daec5c03943f478abc5cb909b208aed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:09.373500 20141 tablet_service.cc:1467] Processing CreateTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd (DEFAULT_TABLE table=test-table [id=daec5c03943f478abc5cb909b208aed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:09.373854 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbcd45a86a74e1da9c22f114be1cdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:09.373853 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbcd45a86a74e1da9c22f114be1cdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:09.374146 20224 log.cc:824] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:09.374890 20224 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: No bootstrap required, opened a new log
I20260925 15:29:09.374977 20224 ts_tablet_manager.cc:1397] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:29:09.376581 20224 raft_consensus.cc:348] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.376785 20224 raft_consensus.cc:374] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:09.376829 20224 raft_consensus.cc:729] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38819747e02f4d478deac6a4708bbdf6, State: Initialized, Role: FOLLOWER
I20260925 15:29:09.376972 20224 consensus_queue.cc:260] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.377280 20226 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: Bootstrap starting.
I20260925 15:29:09.377281 20227 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Bootstrap starting.
I20260925 15:29:09.377307 20224 ts_tablet_manager.cc:1428] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:29:09.377427 20073 heartbeater.cc:503] Master 127.1.163.126:34965 was elected leader, sending a full tablet report...
I20260925 15:29:09.377966 20227 tablet_bootstrap.cc:654] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:09.377966 20226 tablet_bootstrap.cc:654] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:09.378592 20226 log.cc:824] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:09.378582 20227 log.cc:824] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:09.379455 20226 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: No bootstrap required, opened a new log
I20260925 15:29:09.379537 20226 ts_tablet_manager.cc:1397] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:29:09.379552 20227 tablet_bootstrap.cc:492] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: No bootstrap required, opened a new log
I20260925 15:29:09.379623 20227 ts_tablet_manager.cc:1397] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:29:09.381654 20226 raft_consensus.cc:348] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.381654 20227 raft_consensus.cc:348] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.381848 20226 raft_consensus.cc:374] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:09.381848 20227 raft_consensus.cc:374] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:09.381888 20227 raft_consensus.cc:729] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cecb9a5889824db0a81b483aa449df3f, State: Initialized, Role: FOLLOWER
I20260925 15:29:09.381888 20226 raft_consensus.cc:729] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3d0bdc35594ce599fbd9338f71a00e, State: Initialized, Role: FOLLOWER
I20260925 15:29:09.382011 20226 consensus_queue.cc:260] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.382011 20227 consensus_queue.cc:260] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.382313 20226 ts_tablet_manager.cc:1428] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:09.382313 20227 ts_tablet_manager.cc:1428] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:09.382339 20207 heartbeater.cc:503] Master 127.1.163.126:34965 was elected leader, sending a full tablet report...
I20260925 15:29:09.382419 19939 heartbeater.cc:503] Master 127.1.163.126:34965 was elected leader, sending a full tablet report...
W20260925 15:29:09.428555 20074 tablet.cc:2369] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:09.455407 20228 raft_consensus.cc:482] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:09.455543 20228 raft_consensus.cc:504] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.456036 20228 leader_election.cc:288] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e3d0bdc35594ce599fbd9338f71a00e (127.1.163.65:33035), cecb9a5889824db0a81b483aa449df3f (127.1.163.67:40771)
I20260925 15:29:09.459251 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.456262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33035 (local address 127.1.163.66:52295)
0925 15:29:09.456344 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.456357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.456423 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.456667 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.456668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.456670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:09.456760 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.456762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.457509 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.457510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.458385 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.458387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.458962 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.458963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.459048 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.459058 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.459124 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.459175 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:29:09.459268 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.456318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:52295 (local address 127.1.163.65:33035)
0925 15:29:09.456361 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.456363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.456428 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:29:09.456488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.456489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.456499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.456614 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:09.456880 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.457419 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.458483 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.458891 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.458948 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.458956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.459135 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.459195 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:29:09.459473 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.456520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:42185 (local address 127.1.163.67:40771)
0925 15:29:09.456613 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.456616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.456752 (+ 136us) server_negotiation.cc:407] Connection header received
0925 15:29:09.456784 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.456785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.456796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.456858 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:09.457042 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.457606 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.458557 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.459105 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.459171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.459179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.459336 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.459405 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:29:09.459488 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.456446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:40771 (local address 127.1.163.66:42185)
0925 15:29:09.456654 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.456664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.456717 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.456874 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.456875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.456877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:09.456946 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.456947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.457693 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.457693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.458468 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.458471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.459185 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.459185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.459254 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.459258 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.459318 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.459374 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"thread_start_us":139,"threads_started":1}
I20260925 15:29:09.459676 19893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbcd45a86a74e1da9c22f114be1cdbd" candidate_uuid: "38819747e02f4d478deac6a4708bbdf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" is_pre_election: true
I20260925 15:29:09.459925 19893 raft_consensus.cc:2454] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38819747e02f4d478deac6a4708bbdf6 in term 0.
I20260925 15:29:09.459896 20161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbcd45a86a74e1da9c22f114be1cdbd" candidate_uuid: "38819747e02f4d478deac6a4708bbdf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cecb9a5889824db0a81b483aa449df3f" is_pre_election: true
I20260925 15:29:09.460022 20161 raft_consensus.cc:2454] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38819747e02f4d478deac6a4708bbdf6 in term 0.
I20260925 15:29:09.460203 19962 leader_election.cc:302] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38819747e02f4d478deac6a4708bbdf6, 8e3d0bdc35594ce599fbd9338f71a00e; no voters:
I20260925 15:29:09.460357 20228 raft_consensus.cc:2781] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:09.460441 20228 raft_consensus.cc:482] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:09.460460 20228 raft_consensus.cc:3037] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:09.461427 20228 raft_consensus.cc:504] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.461580 20228 leader_election.cc:288] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [CANDIDATE]: Term 1 election: Requested vote from peers 8e3d0bdc35594ce599fbd9338f71a00e (127.1.163.65:33035), cecb9a5889824db0a81b483aa449df3f (127.1.163.67:40771)
I20260925 15:29:09.461719 19893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbcd45a86a74e1da9c22f114be1cdbd" candidate_uuid: "38819747e02f4d478deac6a4708bbdf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d0bdc35594ce599fbd9338f71a00e"
I20260925 15:29:09.461753 20161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbcd45a86a74e1da9c22f114be1cdbd" candidate_uuid: "38819747e02f4d478deac6a4708bbdf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cecb9a5889824db0a81b483aa449df3f"
I20260925 15:29:09.461804 19893 raft_consensus.cc:3037] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:09.461866 20161 raft_consensus.cc:3037] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:09.462473 19893 raft_consensus.cc:2454] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38819747e02f4d478deac6a4708bbdf6 in term 1.
I20260925 15:29:09.462626 20161 raft_consensus.cc:2454] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38819747e02f4d478deac6a4708bbdf6 in term 1.
I20260925 15:29:09.462643 19962 leader_election.cc:302] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38819747e02f4d478deac6a4708bbdf6, 8e3d0bdc35594ce599fbd9338f71a00e; no voters:
I20260925 15:29:09.462718 20228 raft_consensus.cc:2781] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:09.462874 20228 raft_consensus.cc:686] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 1 LEADER]: Becoming Leader. State: Replica: 38819747e02f4d478deac6a4708bbdf6, State: Running, Role: LEADER
I20260925 15:29:09.462986 20228 consensus_queue.cc:237] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } }
I20260925 15:29:09.463846 19748 catalog_manager.cc:5495] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38819747e02f4d478deac6a4708bbdf6 (127.1.163.66). New cstate: current_term: 1 leader_uuid: "38819747e02f4d478deac6a4708bbdf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38819747e02f4d478deac6a4708bbdf6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 } health_report { overall_health: UNKNOWN } } }
I20260925 15:29:09.489269 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.486988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.1.163.66:46853)
0925 15:29:09.487050 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.487052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.487061 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:09.487110 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.487111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.487131 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.487198 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:09.487313 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.487803 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.488590 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.488790 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.488841 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.488976 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:09.489061 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:09.489131 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.489178 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.489188 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:29:09.491133 19893 raft_consensus.cc:1261] T ebbcd45a86a74e1da9c22f114be1cdbd P 8e3d0bdc35594ce599fbd9338f71a00e [term 1 FOLLOWER]: Refusing update from remote peer 38819747e02f4d478deac6a4708bbdf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:09.491144 20161 raft_consensus.cc:1261] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 1 FOLLOWER]: Refusing update from remote peer 38819747e02f4d478deac6a4708bbdf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:09.491494 20228 consensus_queue.cc:1035] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e3d0bdc35594ce599fbd9338f71a00e" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:09.491642 20234 consensus_queue.cc:1035] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cecb9a5889824db0a81b483aa449df3f" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:09.494681 20238 mvcc.cc:205] Tried to move back new op lower bound from 7333274212312932352 to 7333274212201013248. Current Snapshot: MvccSnapshot[applied={T|T < 7333274212312932352}]
I20260925 15:29:09.495289 20239 mvcc.cc:205] Tried to move back new op lower bound from 7333274212312932352 to 7333274212201013248. Current Snapshot: MvccSnapshot[applied={T|T < 7333274212312932352}]
I20260925 15:29:09.497053 19745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:09.497169 19745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:09.498054 19745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75986b8a9d2d41da887925b2b0d7d10e: Sending DeleteTablet for 3 replicas of tablet ebbcd45a86a74e1da9c22f114be1cdbd
I20260925 15:29:09.498476 20007 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:09 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:42032
I20260925 15:29:09.498497 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:09 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:47044
I20260925 15:29:09.498725 19873 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbcd45a86a74e1da9c22f114be1cdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:09 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:48870
I20260925 15:29:09.498909 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19810
I20260925 15:29:09.498989 20247 tablet_replica.cc:330] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: stopping tablet replica
I20260925 15:29:09.499092 20247 raft_consensus.cc:2229] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:09.499197 20247 raft_consensus.cc:2258] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:09.499343 20248 tablet_replica.cc:330] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: stopping tablet replica
I20260925 15:29:09.499425 20248 raft_consensus.cc:2229] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:09.499547 20248 raft_consensus.cc:2258] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:09.501207 20248 ts_tablet_manager.cc:1905] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:09.502058 20247 ts_tablet_manager.cc:1905] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:09.503355 20248 ts_tablet_manager.cc:1918] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:09.503512 20248 log.cc:1196] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/ebbcd45a86a74e1da9c22f114be1cdbd
I20260925 15:29:09.503887 20248 ts_tablet_manager.cc:1939] T ebbcd45a86a74e1da9c22f114be1cdbd P 38819747e02f4d478deac6a4708bbdf6: Deleting consensus metadata
I20260925 15:29:09.504181 20247 ts_tablet_manager.cc:1918] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:09.504261 20247 log.cc:1196] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/ebbcd45a86a74e1da9c22f114be1cdbd
I20260925 15:29:09.504528 19735 catalog_manager.cc:4841] TS 38819747e02f4d478deac6a4708bbdf6 (127.1.163.66:46853): tablet ebbcd45a86a74e1da9c22f114be1cdbd (table test-table [id=daec5c03943f478abc5cb909b208aed5]) successfully deleted
I20260925 15:29:09.504583 20247 ts_tablet_manager.cc:1939] T ebbcd45a86a74e1da9c22f114be1cdbd P cecb9a5889824db0a81b483aa449df3f: Deleting consensus metadata
I20260925 15:29:09.505031 19732 catalog_manager.cc:4841] TS cecb9a5889824db0a81b483aa449df3f (127.1.163.67:40771): tablet ebbcd45a86a74e1da9c22f114be1cdbd (table test-table [id=daec5c03943f478abc5cb909b208aed5]) successfully deleted
W20260925 15:29:09.505950 19734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:33035 (error 108)
I20260925 15:29:09.506505 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.506358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:33035 (local address 127.0.0.1:48882)
0925 15:29:09.506411 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.506465 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:29:09.506525 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19944
W20260925 15:29:09.506626 19734 catalog_manager.cc:4584] TS 8e3d0bdc35594ce599fbd9338f71a00e (127.1.163.65:33035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebbcd45a86a74e1da9c22f114be1cdbd: Network error: Client connection negotiation failed: client connection to 127.1.163.65:33035: connect: Connection refused (error 111)
I20260925 15:29:09.512128 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20078
I20260925 15:29:09.517236 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 19716
2026-09-25T15:29:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:09Z Disabled control of system clock
I20260925 15:29:09.546134 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34989
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:09.628507 20254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:09.628777 20254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:09.628799 20254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:09.630112 20254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:09.630155 20254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:09.630168 20254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:09.630182 20254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:09.630193 20254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:09.631593 20254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:34989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:09.631868 20254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:09.632200 20254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:09.637308 20260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.637351 20259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.637354 20262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.637544 20254 server_base.cc:1034] running on GCE node
I20260925 15:29:09.637797 20254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:09.638113 20254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:09.639293 20254 hybrid_clock.cc:648] HybridClock initialized: now 1790350149639286 us; error 36 us; skew 500 ppm
I20260925 15:29:09.640779 20254 webserver.cc:460] Webserver started at http://127.1.163.126:40617/ using document root <none> and password file <none>
I20260925 15:29:09.641062 20254 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:09.641135 20254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:09.641291 20254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:09.642407 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "4e4b52759a22493baf3d2983b0fc60c6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "19ce373ff3373c3312e6c6393d41288c"
server_key_iv: "2a3f6ca8bd00052b4c9bb10ba1557516"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.642787 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "4e4b52759a22493baf3d2983b0fc60c6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "19ce373ff3373c3312e6c6393d41288c"
server_key_iv: "2a3f6ca8bd00052b4c9bb10ba1557516"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.644198 20254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:09.645123 20268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.645308 20254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:29:09.645412 20254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "4e4b52759a22493baf3d2983b0fc60c6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "19ce373ff3373c3312e6c6393d41288c"
server_key_iv: "2a3f6ca8bd00052b4c9bb10ba1557516"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.645498 20254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.663530 20254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.663903 20254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.664083 20254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.669340 20254 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:34989
I20260925 15:29:09.669377 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:34989 every 8 connection(s)
I20260925 15:29:09.669744 20254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:09.670710 20321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:09.670714 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20254
I20260925 15:29:09.670825 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:09.671017 1677 external_mini_cluster.cc:1428] Setting key 33e41d15d91d161938ccec13176b02a6
I20260925 15:29:09.673918 20321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: Bootstrap starting.
I20260925 15:29:09.674623 20321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:09.675047 20321 log.cc:824] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:09.675866 20321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: No bootstrap required, opened a new log
I20260925 15:29:09.677137 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.671582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.1.163.126:34989)
0925 15:29:09.671781 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.671788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.671812 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:29:09.672257 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.672264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.672325 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.672441 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.672707 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.673577 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.674675 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.675337 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.675503 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.676340 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.676345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.676345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:09.676596 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.676642 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:09.676656 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.676727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.676820 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.676905 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260925 15:29:09.678181 20321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } }
I20260925 15:29:09.678319 20321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:09.678341 20321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4b52759a22493baf3d2983b0fc60c6, State: Initialized, Role: FOLLOWER
I20260925 15:29:09.678491 20321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } }
I20260925 15:29:09.678565 20321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:09.678593 20321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:09.678644 20321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:09.679239 20321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } }
I20260925 15:29:09.679360 20321 leader_election.cc:302] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e4b52759a22493baf3d2983b0fc60c6; no voters:
I20260925 15:29:09.679556 20321 leader_election.cc:288] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:09.679631 20326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:09.679792 20326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4b52759a22493baf3d2983b0fc60c6, State: Running, Role: LEADER
I20260925 15:29:09.679822 20321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:09.679899 20326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } }
I20260925 15:29:09.680244 20327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e4b52759a22493baf3d2983b0fc60c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } } }
I20260925 15:29:09.680334 20327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:09.680254 20328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4b52759a22493baf3d2983b0fc60c6. Latest consensus state: current_term: 1 leader_uuid: "4e4b52759a22493baf3d2983b0fc60c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4b52759a22493baf3d2983b0fc60c6" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 34989 } } }
I20260925 15:29:09.680384 20328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:09.680631 20333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:09.681329 20333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:09.682842 20333 catalog_manager.cc:1347] Generated new cluster ID: 0ddc1f2f63e24d0c9b5d086ea3433271
I20260925 15:29:09.682894 20333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:09.697834 20333 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:09.698591 20333 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:09.704311 20333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: Generated new TSK 0
I20260925 15:29:09.704519 20333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:09.705891 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--builtin_ntp_servers=127.1.163.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:09.798749 20345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:09.798954 20345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:09.798976 20345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:09.800145 20345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:09.800189 20345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:09.800215 20345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:09.801640 20345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:09.801888 20345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:09.802204 20345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:09.807688 20350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.807655 20351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.807665 20353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.808043 20345 server_base.cc:1034] running on GCE node
I20260925 15:29:09.808255 20345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:09.808563 20345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:09.809788 20345 hybrid_clock.cc:648] HybridClock initialized: now 1790350149809774 us; error 27 us; skew 500 ppm
I20260925 15:29:09.811276 20345 webserver.cc:460] Webserver started at http://127.1.163.65:39409/ using document root <none> and password file <none>
I20260925 15:29:09.811573 20345 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:09.811643 20345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:09.811779 20345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:09.813176 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "10bc19ddae0b25353205a05709cfe4d5"
server_key_iv: "6a3f3e4c705760c53dbeea9f8a39db86"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.813570 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "10bc19ddae0b25353205a05709cfe4d5"
server_key_iv: "6a3f3e4c705760c53dbeea9f8a39db86"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.815145 20345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:09.816031 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.816231 20345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:09.816321 20345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "10bc19ddae0b25353205a05709cfe4d5"
server_key_iv: "6a3f3e4c705760c53dbeea9f8a39db86"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.816403 20345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.844214 20345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.844514 20345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.844710 20345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.844926 20345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:09.845284 20345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:09.845317 20345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.845382 20345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:09.845410 20345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.851403 20345 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:35977
I20260925 15:29:09.851439 20472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:35977 every 8 connection(s)
I20260925 15:29:09.851781 20345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:29:09.852042 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20345
I20260925 15:29:09.852160 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:09.852403 1677 external_mini_cluster.cc:1428] Setting key 3a9633f784210f1f182f8a7d23e5ceff
I20260925 15:29:09.853521 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--builtin_ntp_servers=127.1.163.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:09.857872 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.852736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:39979 (local address 127.1.163.126:34989)
0925 15:29:09.852809 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.852811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.853133 (+ 322us) server_negotiation.cc:407] Connection header received
0925 15:29:09.853773 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.853774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.853790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.853845 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.854557 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.855072 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.856090 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.856411 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.856482 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.857514 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.857516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.857516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:09.857523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.857545 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:09.857550 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.857619 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.857707 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.857743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:29:09.858013 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.852737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34989 (local address 127.1.163.65:39979)
0925 15:29:09.852975 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.853023 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.853631 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.853929 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.853933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.853950 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:09.854462 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.854467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.855175 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.855176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.855994 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.855996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.856504 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.856505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.856704 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.857324 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:09.857325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:09.857376 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:09.857410 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:09.857625 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:09.857630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:09.857632 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.857688 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.857775 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260925 15:29:09.858481 20473 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34989
I20260925 15:29:09.858597 20473 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:09.858825 20473 heartbeater.cc:511] Master 127.1.163.126:34989 requested a full tablet report, sending...
I20260925 15:29:09.859335 20282 ts_manager.cc:194] Registered new tserver with Master: d77c6aaf41a743f6beb90d86f7ecfe9d (127.1.163.65:35977)
I20260925 15:29:09.859957 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.65:39979
W20260925 15:29:09.938648 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:09.938859 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:09.938882 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:09.940112 20476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:09.940160 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:09.940186 20476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:09.941675 20476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:09.941955 20476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:09.942274 20476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:09.947685 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.947690 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:09.947674 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:09.947753 20476 server_base.cc:1034] running on GCE node
I20260925 15:29:09.948112 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:09.948436 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:09.949584 20476 hybrid_clock.cc:648] HybridClock initialized: now 1790350149949584 us; error 41 us; skew 500 ppm
I20260925 15:29:09.951292 20476 webserver.cc:460] Webserver started at http://127.1.163.66:36659/ using document root <none> and password file <none>
I20260925 15:29:09.951578 20476 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:09.951627 20476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:09.951777 20476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:09.952994 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "d5bb040814253c701879206c5d6f4500"
server_key_iv: "9854aeeb27f104cafb81703b395a6e6d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.953368 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "d5bb040814253c701879206c5d6f4500"
server_key_iv: "9854aeeb27f104cafb81703b395a6e6d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.954708 20476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:09.955416 20490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.955577 20476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:29:09.955684 20476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6"
format_stamp: "Formatted at 2026-09-25 15:29:09 on dist-test-slave-hvv5"
server_key: "d5bb040814253c701879206c5d6f4500"
server_key_iv: "9854aeeb27f104cafb81703b395a6e6d"
server_key_version: "encryptionkey@0"
I20260925 15:29:09.955763 20476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:09.969954 20476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:09.970292 20476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:09.970455 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:09.970687 20476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:09.971066 20476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:09.971124 20476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.971179 20476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:09.971208 20476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:09.977025 20476 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39597
I20260925 15:29:09.977068 20603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39597 every 8 connection(s)
I20260925 15:29:09.977439 20476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:29:09.979370 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20476
I20260925 15:29:09.979488 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:09.979686 1677 external_mini_cluster.cc:1428] Setting key ff912e223e0f165a32530a4677456f2a
I20260925 15:29:09.980731 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--builtin_ntp_servers=127.1.163.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:09.983544 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.978195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:56085 (local address 127.1.163.126:34989)
0925 15:29:09.978258 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:09.978260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:09.978505 (+ 245us) server_negotiation.cc:407] Connection header received
0925 15:29:09.979081 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:09.979082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:09.979095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:09.979148 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:09.980038 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.980881 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.981958 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.982176 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.982257 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.983234 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:09.983235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:09.983235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:09.983240 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:09.983259 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:09.983263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:09.983324 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:09.983394 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:29:09.983421 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:29:09.983708 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:09.978187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34989 (local address 127.1.163.66:56085)
0925 15:29:09.978383 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:09.978422 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:09.978974 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:09.979242 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:09.979245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:09.979253 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:09.979936 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.979942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.980972 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.980973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.981844 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:09.981847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:09.982265 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:09.982267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:09.982423 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:09.983084 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:09.983084 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:09.983113 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:09.983136 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:09.983332 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:09.983336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:09.983338 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:09.983387 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:29:09.983493 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260925 15:29:09.984315 20604 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34989
I20260925 15:29:09.984431 20604 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:09.984676 20604 heartbeater.cc:511] Master 127.1.163.126:34989 requested a full tablet report, sending...
I20260925 15:29:09.985095 20282 ts_manager.cc:194] Registered new tserver with Master: cad0faecbe1e4fc7862cfd0faaf3eab6 (127.1.163.66:39597)
I20260925 15:29:09.985478 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.66:56085
W20260925 15:29:10.065609 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:10.065836 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:10.065860 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:10.067090 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:10.067138 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:10.067165 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:10.068660 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:10.068936 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:10.069271 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:10.074651 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:10.074703 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:10.074682 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:10.074791 20607 server_base.cc:1034] running on GCE node
I20260925 15:29:10.075125 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:10.075394 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:10.076591 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790350150076573 us; error 51 us; skew 500 ppm
I20260925 15:29:10.077864 20607 webserver.cc:460] Webserver started at http://127.1.163.67:46751/ using document root <none> and password file <none>
I20260925 15:29:10.078132 20607 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:10.078202 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:10.078358 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:10.079502 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "b7b0ea4443be464589e43b8d5c4f87df"
format_stamp: "Formatted at 2026-09-25 15:29:10 on dist-test-slave-hvv5"
server_key: "a7527df15c03ade9aded097834c53512"
server_key_iv: "10a624d43330f6d14d7ddeb0b1f4af1d"
server_key_version: "encryptionkey@0"
I20260925 15:29:10.079913 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "b7b0ea4443be464589e43b8d5c4f87df"
format_stamp: "Formatted at 2026-09-25 15:29:10 on dist-test-slave-hvv5"
server_key: "a7527df15c03ade9aded097834c53512"
server_key_iv: "10a624d43330f6d14d7ddeb0b1f4af1d"
server_key_version: "encryptionkey@0"
I20260925 15:29:10.081458 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:10.082183 20621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:10.082350 20607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:29:10.082471 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "b7b0ea4443be464589e43b8d5c4f87df"
format_stamp: "Formatted at 2026-09-25 15:29:10 on dist-test-slave-hvv5"
server_key: "a7527df15c03ade9aded097834c53512"
server_key_iv: "10a624d43330f6d14d7ddeb0b1f4af1d"
server_key_version: "encryptionkey@0"
I20260925 15:29:10.082548 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:10.100093 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:10.100453 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:10.100641 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:10.100895 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:10.101243 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:10.101297 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:10.101341 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:10.101366 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:10.107406 20607 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:36935
I20260925 15:29:10.107456 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:36935 every 8 connection(s)
I20260925 15:29:10.107803 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:29:10.113466 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.108837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:46479 (local address 127.1.163.126:34989)
0925 15:29:10.108923 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.108926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.109121 (+ 195us) server_negotiation.cc:407] Connection header received
0925 15:29:10.109682 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.109684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.109698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.109745 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:10.110441 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.110943 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.111960 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.112156 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.112236 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.113153 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:10.113154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:10.113154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:10.113159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:10.113182 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:10.113187 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:10.113248 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.113345 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.113374 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:29:10.113715 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.108727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:34989 (local address 127.1.163.67:46479)
0925 15:29:10.108988 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.109034 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.109566 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.109815 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.109819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.109829 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:10.110342 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.110347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.111040 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.111041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.111840 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.111843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.112243 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.112244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.112389 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.112987 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:10.112987 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:10.113023 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:10.113049 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:10.113249 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:10.113255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:10.113259 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.113348 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.113459 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
I20260925 15:29:10.114205 20735 heartbeater.cc:347] Connected to a master server at 127.1.163.126:34989
I20260925 15:29:10.114300 20735 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:10.114456 20735 heartbeater.cc:511] Master 127.1.163.126:34989 requested a full tablet report, sending...
I20260925 15:29:10.114806 20282 ts_manager.cc:194] Registered new tserver with Master: b7b0ea4443be464589e43b8d5c4f87df (127.1.163.67:36935)
I20260925 15:29:10.115214 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.67:46479
I20260925 15:29:10.116483 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20607
I20260925 15:29:10.116634 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:10.116793 1677 external_mini_cluster.cc:1428] Setting key 8d7857db762987c387c723521eef1f38
I20260925 15:29:10.117615 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:10.121935 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.119006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.1.163.126:34989)
0925 15:29:10.119044 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.119046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.119150 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:29:10.119183 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.119184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.119192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.119238 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:10.119363 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.119946 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.120745 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.121036 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.121091 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.121712 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:10.121713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:10.121713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:10.121717 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:10.121731 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:10.121734 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:10.121805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.121869 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.121877 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:29:10.123472 20282 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:10.125597 20282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:10.132144 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.128181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35977 (local address 127.0.0.1:41796)
0925 15:29:10.128735 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.128746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.128786 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.129070 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.129071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.129073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:10.129123 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.129125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.130269 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.130270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.131164 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.131167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.131847 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.131848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.131920 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.131935 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.132011 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.132065 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":80,"threads_started":1}
I20260925 15:29:10.132159 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.128434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.1.163.65:35977)
0925 15:29:10.128923 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.128928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.128937 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:10.128977 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.128979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.128993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.129071 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:10.129187 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.130190 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.131256 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.131748 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.131843 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.131949 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.132026 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.132064 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260925 15:29:10.132494 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.128052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.1.163.67:36935)
0925 15:29:10.128225 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.128229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.128502 (+ 273us) server_negotiation.cc:407] Connection header received
0925 15:29:10.128585 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.128587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.128602 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.128674 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:10.128845 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.129975 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.131297 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.132023 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.132096 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.132179 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.132369 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.132421 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260925 15:29:10.132364 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.127984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39597 (local address 127.0.0.1:48248)
0925 15:29:10.128308 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.128330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.128409 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.128662 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.128665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.128668 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:10.128828 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.128830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.129827 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.129828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.131186 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.131188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.132128 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.132129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.132192 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.132197 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.132229 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.132271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":119,"thread_start_us":98,"threads_started":1}
I20260925 15:29:10.132752 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.128099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.1.163.66:39597)
0925 15:29:10.128450 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.128455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.128469 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:29:10.128510 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.128512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.128528 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.128631 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:10.128893 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.129716 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.131297 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.132023 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.132449 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.132569 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.132601 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.132647 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":84,"threads_started":1}
I20260925 15:29:10.132467 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.127937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:36935 (local address 127.0.0.1:54808)
0925 15:29:10.128287 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.128330 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.128408 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.128670 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.128670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.128672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:10.128753 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.128755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.130080 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.130081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.131186 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.131188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.132209 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.132210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.132277 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.132279 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.132333 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.132366 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":9,"thread_start_us":98,"threads_started":1}
I20260925 15:29:10.133550 20538 tablet_service.cc:1467] Processing CreateTablet for tablet d560832b5a664329a7b4e10ef4cf53ba (DEFAULT_TABLE table=test-table [id=a0c2f20a75354985b393223b2afe9d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:10.133592 20669 tablet_service.cc:1467] Processing CreateTablet for tablet d560832b5a664329a7b4e10ef4cf53ba (DEFAULT_TABLE table=test-table [id=a0c2f20a75354985b393223b2afe9d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:10.133555 20407 tablet_service.cc:1467] Processing CreateTablet for tablet d560832b5a664329a7b4e10ef4cf53ba (DEFAULT_TABLE table=test-table [id=a0c2f20a75354985b393223b2afe9d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:10.133924 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d560832b5a664329a7b4e10ef4cf53ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:10.133924 20669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d560832b5a664329a7b4e10ef4cf53ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:10.133924 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d560832b5a664329a7b4e10ef4cf53ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:10.136669 20749 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: Bootstrap starting.
I20260925 15:29:10.137277 20750 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Bootstrap starting.
I20260925 15:29:10.137473 20749 tablet_bootstrap.cc:654] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:10.137825 20749 log.cc:824] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:10.137989 20750 tablet_bootstrap.cc:654] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:10.138276 20750 log.cc:824] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:10.138622 20751 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: Bootstrap starting.
I20260925 15:29:10.138715 20749 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: No bootstrap required, opened a new log
I20260925 15:29:10.138783 20749 ts_tablet_manager.cc:1397] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:10.139060 20750 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: No bootstrap required, opened a new log
I20260925 15:29:10.139146 20750 ts_tablet_manager.cc:1397] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:10.139261 20751 tablet_bootstrap.cc:654] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:10.139590 20751 log.cc:824] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:10.140584 20751 tablet_bootstrap.cc:492] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: No bootstrap required, opened a new log
I20260925 15:29:10.140653 20751 ts_tablet_manager.cc:1397] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:10.141480 20749 raft_consensus.cc:348] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.141605 20749 raft_consensus.cc:374] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:10.141626 20749 raft_consensus.cc:729] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad0faecbe1e4fc7862cfd0faaf3eab6, State: Initialized, Role: FOLLOWER
I20260925 15:29:10.141752 20749 consensus_queue.cc:260] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.142036 20749 ts_tablet_manager.cc:1428] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:10.142064 20604 heartbeater.cc:503] Master 127.1.163.126:34989 was elected leader, sending a full tablet report...
I20260925 15:29:10.142063 20750 raft_consensus.cc:348] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.142211 20750 raft_consensus.cc:374] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:10.142243 20750 raft_consensus.cc:729] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b0ea4443be464589e43b8d5c4f87df, State: Initialized, Role: FOLLOWER
I20260925 15:29:10.142349 20750 consensus_queue.cc:260] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.142599 20750 ts_tablet_manager.cc:1428] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:29:10.142802 20735 heartbeater.cc:503] Master 127.1.163.126:34989 was elected leader, sending a full tablet report...
I20260925 15:29:10.143270 20751 raft_consensus.cc:348] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.143419 20751 raft_consensus.cc:374] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:10.143441 20751 raft_consensus.cc:729] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77c6aaf41a743f6beb90d86f7ecfe9d, State: Initialized, Role: FOLLOWER
I20260925 15:29:10.143517 20751 consensus_queue.cc:260] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.143682 20751 ts_tablet_manager.cc:1428] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:29:10.143812 20473 heartbeater.cc:503] Master 127.1.163.126:34989 was elected leader, sending a full tablet report...
I20260925 15:29:10.183703 20756 raft_consensus.cc:482] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:10.183828 20756 raft_consensus.cc:504] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.184271 20756 leader_election.cc:288] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad0faecbe1e4fc7862cfd0faaf3eab6 (127.1.163.66:39597), d77c6aaf41a743f6beb90d86f7ecfe9d (127.1.163.65:35977)
I20260925 15:29:10.188393 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.184583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:46503 (local address 127.1.163.65:35977)
0925 15:29:10.184691 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.184693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.184789 (+ 96us) server_negotiation.cc:407] Connection header received
0925 15:29:10.184865 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.184865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.184876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.184932 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:10.185094 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.185839 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.187417 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.187995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.188083 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.188092 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.188292 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.188334 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:29:10.188385 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.184582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:34531 (local address 127.1.163.66:39597)
0925 15:29:10.184633 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.184635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.184645 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:10.184733 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.184734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.184743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.184819 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:10.185037 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.185827 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.187422 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.188029 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.188094 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.188102 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.188270 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.188314 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:29:10.188403 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.184480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39597 (local address 127.1.163.67:34531)
0925 15:29:10.184575 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.184587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.184636 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.184836 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.184837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.184840 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:10.184953 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.184956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.185936 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.185937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.187311 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.187315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.188105 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.188106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.188187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.188191 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.188252 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.188306 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":79}
I20260925 15:29:10.188431 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.184480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35977 (local address 127.1.163.67:46503)
0925 15:29:10.184745 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.184752 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:10.184809 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:10.184953 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:10.184954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:10.184956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:10.185017 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.185019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.185919 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.185920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.187321 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:10.187323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.188105 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.188106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:10.188187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.188191 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:10.188252 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:29:10.188305 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":107,"thread_start_us":122,"threads_started":1}
I20260925 15:29:10.188789 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d560832b5a664329a7b4e10ef4cf53ba" candidate_uuid: "b7b0ea4443be464589e43b8d5c4f87df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" is_pre_election: true
I20260925 15:29:10.188962 20558 raft_consensus.cc:2454] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b0ea4443be464589e43b8d5c4f87df in term 0.
I20260925 15:29:10.189189 20624 leader_election.cc:302] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7b0ea4443be464589e43b8d5c4f87df, cad0faecbe1e4fc7862cfd0faaf3eab6; no voters:
I20260925 15:29:10.189185 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d560832b5a664329a7b4e10ef4cf53ba" candidate_uuid: "b7b0ea4443be464589e43b8d5c4f87df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" is_pre_election: true
I20260925 15:29:10.189338 20427 raft_consensus.cc:2454] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b0ea4443be464589e43b8d5c4f87df in term 0.
I20260925 15:29:10.189347 20756 raft_consensus.cc:2781] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:10.189409 20756 raft_consensus.cc:482] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:10.189442 20756 raft_consensus.cc:3037] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:10.190230 20756 raft_consensus.cc:504] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.190393 20756 leader_election.cc:288] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [CANDIDATE]: Term 1 election: Requested vote from peers cad0faecbe1e4fc7862cfd0faaf3eab6 (127.1.163.66:39597), d77c6aaf41a743f6beb90d86f7ecfe9d (127.1.163.65:35977)
I20260925 15:29:10.190526 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d560832b5a664329a7b4e10ef4cf53ba" candidate_uuid: "b7b0ea4443be464589e43b8d5c4f87df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6"
I20260925 15:29:10.190555 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d560832b5a664329a7b4e10ef4cf53ba" candidate_uuid: "b7b0ea4443be464589e43b8d5c4f87df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d"
I20260925 15:29:10.190611 20558 raft_consensus.cc:3037] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:10.190618 20427 raft_consensus.cc:3037] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:10.191536 20427 raft_consensus.cc:2454] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b0ea4443be464589e43b8d5c4f87df in term 1.
I20260925 15:29:10.191536 20558 raft_consensus.cc:2454] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7b0ea4443be464589e43b8d5c4f87df in term 1.
I20260925 15:29:10.191736 20622 leader_election.cc:302] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7b0ea4443be464589e43b8d5c4f87df, d77c6aaf41a743f6beb90d86f7ecfe9d; no voters:
I20260925 15:29:10.191972 20756 raft_consensus.cc:2781] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:10.192114 20756 raft_consensus.cc:686] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 1 LEADER]: Becoming Leader. State: Replica: b7b0ea4443be464589e43b8d5c4f87df, State: Running, Role: LEADER
I20260925 15:29:10.192211 20756 consensus_queue.cc:237] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } }
I20260925 15:29:10.193372 20282 catalog_manager.cc:5495] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df reported cstate change: term changed from 0 to 1, leader changed from <none> to b7b0ea4443be464589e43b8d5c4f87df (127.1.163.67). New cstate: current_term: 1 leader_uuid: "b7b0ea4443be464589e43b8d5c4f87df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b0ea4443be464589e43b8d5c4f87df" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 36935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 } health_report { overall_health: UNKNOWN } } }
I20260925 15:29:10.207296 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.204351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.1.163.67:36935)
0925 15:29:10.204445 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:10.204447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:10.204458 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:10.204520 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:10.204521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:10.204582 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:10.204666 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:10.204899 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.205592 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.206378 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:10.206586 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:10.206645 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:10.206823 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:10.206919 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:10.207004 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:10.207129 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:29:10.207165 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:29:10.209302 20558 raft_consensus.cc:1261] T d560832b5a664329a7b4e10ef4cf53ba P cad0faecbe1e4fc7862cfd0faaf3eab6 [term 1 FOLLOWER]: Refusing update from remote peer b7b0ea4443be464589e43b8d5c4f87df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:10.209304 20427 raft_consensus.cc:1261] T d560832b5a664329a7b4e10ef4cf53ba P d77c6aaf41a743f6beb90d86f7ecfe9d [term 1 FOLLOWER]: Refusing update from remote peer b7b0ea4443be464589e43b8d5c4f87df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:10.209849 20759 consensus_queue.cc:1035] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad0faecbe1e4fc7862cfd0faaf3eab6" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:10.210014 20756 consensus_queue.cc:1035] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77c6aaf41a743f6beb90d86f7ecfe9d" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:10.212662 20762 mvcc.cc:205] Tried to move back new op lower bound from 7333274215254601728 to 7333274215187992576. Current Snapshot: MvccSnapshot[applied={T|T < 7333274215254601728}]
I20260925 15:29:10.213440 20763 mvcc.cc:205] Tried to move back new op lower bound from 7333274215254601728 to 7333274215187992576. Current Snapshot: MvccSnapshot[applied={T|T < 7333274215254601728}]
I20260925 15:29:10.214764 20766 mvcc.cc:205] Tried to move back new op lower bound from 7333274215254601728 to 7333274215187992576. Current Snapshot: MvccSnapshot[applied={T|T < 7333274215254601728}]
I20260925 15:29:10.215574 20285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:10.215672 20285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:10.216503 20285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e4b52759a22493baf3d2983b0fc60c6: Sending DeleteTablet for 3 replicas of tablet d560832b5a664329a7b4e10ef4cf53ba
I20260925 15:29:10.216866 20669 tablet_service.cc:1514] Processing DeleteTablet for tablet d560832b5a664329a7b4e10ef4cf53ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:10 UTC) from {username='slave'} at 127.0.0.1:54808
I20260925 15:29:10.216895 20538 tablet_service.cc:1514] Processing DeleteTablet for tablet d560832b5a664329a7b4e10ef4cf53ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:10 UTC) from {username='slave'} at 127.0.0.1:48248
I20260925 15:29:10.216904 20407 tablet_service.cc:1514] Processing DeleteTablet for tablet d560832b5a664329a7b4e10ef4cf53ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:10 UTC) from {username='slave'} at 127.0.0.1:41796
I20260925 15:29:10.217080 20772 tablet_replica.cc:330] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: stopping tablet replica
I20260925 15:29:10.217166 20772 raft_consensus.cc:2229] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:10.217252 20772 raft_consensus.cc:2258] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:10.217300 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20345
I20260925 15:29:10.218482 20772 ts_tablet_manager.cc:1905] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:10.220436 20772 ts_tablet_manager.cc:1918] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:10.220515 20772 log.cc:1196] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/d560832b5a664329a7b4e10ef4cf53ba
I20260925 15:29:10.220824 20772 ts_tablet_manager.cc:1939] T d560832b5a664329a7b4e10ef4cf53ba P b7b0ea4443be464589e43b8d5c4f87df: Deleting consensus metadata
I20260925 15:29:10.221235 20269 catalog_manager.cc:4841] TS b7b0ea4443be464589e43b8d5c4f87df (127.1.163.67:36935): tablet d560832b5a664329a7b4e10ef4cf53ba (table test-table [id=a0c2f20a75354985b393223b2afe9d5e]) successfully deleted
W20260925 15:29:10.225162 20269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:35977 (error 108)
I20260925 15:29:10.225419 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20476
I20260925 15:29:10.225701 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.225578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:35977 (local address 127.0.0.1:41810)
0925 15:29:10.225628 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.225666 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 15:29:10.225802 20269 catalog_manager.cc:4584] TS d77c6aaf41a743f6beb90d86f7ecfe9d (127.1.163.65:35977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d560832b5a664329a7b4e10ef4cf53ba: Network error: Client connection negotiation failed: client connection to 127.1.163.65:35977: connect: Connection refused (error 111)
I20260925 15:29:10.231395 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20607
I20260925 15:29:10.231542 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:10.231403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39597 (local address 127.0.0.1:48250)
0925 15:29:10.231466 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:10.231497 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.66:39597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:29:10.235679 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20254
2026-09-25T15:29:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (717 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:10 dist-test-slave-hvv5 krb5kdc[20778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:10 dist-test-slave-hvv5 krb5kdc[20778](info): set up 2 sockets
Sep 25 15:29:10 dist-test-slave-hvv5 krb5kdc[20778](info): commencing operation
krb5kdc: starting...
W20260925 15:29:11.288744 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:11.411945 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41463
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:11.494632 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:11.494938 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:11.494982 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:11.496212 20791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:11.496284 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:11.496307 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:11.496327 20791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:11.496368 20791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:11.497951 20791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:41463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:11.498250 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:11.498505 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:11.504395 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:11.504448 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:11.504415 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:11.504595 20791 server_base.cc:1034] running on GCE node
I20260925 15:29:11.504999 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:11.505321 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:11.506506 20791 hybrid_clock.cc:648] HybridClock initialized: now 1790350151506477 us; error 45 us; skew 500 ppm
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:11.509109 20791 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:11.509552 20791 webserver.cc:460] Webserver started at http://127.1.163.126:41733/ using document root <none> and password file <none>
I20260925 15:29:11.509809 20791 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:11.509881 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:11.510035 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:11.511163 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "ee165e0e246a4b62b9c3de7d80675549"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "20d6ec5fb08dc0e51c63522ebc258785"
server_key_iv: "fbce01d2d9443979842f3c31ae29ca88"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.511539 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "ee165e0e246a4b62b9c3de7d80675549"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "20d6ec5fb08dc0e51c63522ebc258785"
server_key_iv: "fbce01d2d9443979842f3c31ae29ca88"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.512926 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:11.513742 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.513943 20791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:11.514053 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "ee165e0e246a4b62b9c3de7d80675549"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "20d6ec5fb08dc0e51c63522ebc258785"
server_key_iv: "fbce01d2d9443979842f3c31ae29ca88"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.514137 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:11.533473 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:11.536031 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:11.536228 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:11.540256 20791 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:41463
I20260925 15:29:11.540292 20858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:41463 every 8 connection(s)
I20260925 15:29:11.540678 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:11.541554 20859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:11.543803 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: Bootstrap starting.
I20260925 15:29:11.544413 20859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:11.544734 20859 log.cc:824] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:11.545487 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: No bootstrap required, opened a new log
I20260925 15:29:11.547079 20859 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } }
I20260925 15:29:11.547288 20859 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:11.547334 20859 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee165e0e246a4b62b9c3de7d80675549, State: Initialized, Role: FOLLOWER
I20260925 15:29:11.547370 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20791
I20260925 15:29:11.547492 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:11.547508 20859 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } }
I20260925 15:29:11.547642 20859 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:11.547693 20859 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:11.547710 1677 external_mini_cluster.cc:1428] Setting key 0afcc6759aa7eacf36497804960fadaf
I20260925 15:29:11.547739 20859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:11.548456 20859 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } }
I20260925 15:29:11.548615 20859 leader_election.cc:302] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee165e0e246a4b62b9c3de7d80675549; no voters:
I20260925 15:29:11.548841 20859 leader_election.cc:288] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:11.548893 20864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:11.549057 20864 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [term 1 LEADER]: Becoming Leader. State: Replica: ee165e0e246a4b62b9c3de7d80675549, State: Running, Role: LEADER
I20260925 15:29:11.549109 20859 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:11.549188 20864 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } }
I20260925 15:29:11.549587 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee165e0e246a4b62b9c3de7d80675549. Latest consensus state: current_term: 1 leader_uuid: "ee165e0e246a4b62b9c3de7d80675549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } } }
I20260925 15:29:11.549669 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:11.549623 20865 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee165e0e246a4b62b9c3de7d80675549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee165e0e246a4b62b9c3de7d80675549" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 41463 } } }
I20260925 15:29:11.549861 20865 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:11.550621 20872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:11.551317 20872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:11.552925 20872 catalog_manager.cc:1347] Generated new cluster ID: 3d6be55d4b8d4c9eb18739e1a33c8f6e
I20260925 15:29:11.552977 20872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:11.560048 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:11.548578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.1.163.126:41463)
0925 15:29:11.548803 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:11.548811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:11.548865 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:29:11.548990 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:11.548999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:11.549065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:11.550096 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:11.550230 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.551618 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.552622 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.553276 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.553419 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:11.555898 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:11.555902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:11.555903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:11.555922 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:11.558188 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.558580 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.558583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.558584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.558689 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.558922 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.558922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.558923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.559178 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:11.559364 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:11.559444 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:29:11.559621 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260925 15:29:11.562074 20872 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:11.562700 20872 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:11.568010 20872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: Generated new TSK 0
I20260925 15:29:11.568188 20872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.163.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
Entry for principal kudu/127.1.163.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab.
I20260925 15:29:11.604990 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:11.691157 20885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:11.691378 20885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:11.691406 20885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:11.692678 20885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:11.692723 20885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:11.692750 20885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:11.694355 20885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.65.keytab
--principal=kudu/127.1.163.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:11.694640 20885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:11.694890 20885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:11.700361 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:11.700351 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:11.700352 20885 server_base.cc:1034] running on GCE node
W20260925 15:29:11.700367 20893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:11.700810 20885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:11.701035 20885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:11.702185 20885 hybrid_clock.cc:648] HybridClock initialized: now 1790350151702164 us; error 44 us; skew 500 ppm
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:11.704771 20885 init.cc:377] Logged in from keytab as kudu/127.1.163.65@KRBTEST.COM (short username kudu)
I20260925 15:29:11.705245 20885 webserver.cc:460] Webserver started at http://127.1.163.65:32947/ using document root <none> and password file <none>
I20260925 15:29:11.705515 20885 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:11.705564 20885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:11.705699 20885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:11.706863 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "b05d7d51a68d4274bb2ac21d9a5f2836"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "14fe49d38d45da0cbbb27b8db13b868d"
server_key_iv: "09ac378fcf49968e4edc887b91fa7b7b"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.707257 20885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "b05d7d51a68d4274bb2ac21d9a5f2836"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "14fe49d38d45da0cbbb27b8db13b868d"
server_key_iv: "09ac378fcf49968e4edc887b91fa7b7b"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.708714 20885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:11.709534 20900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.709796 20885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:11.709880 20885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "b05d7d51a68d4274bb2ac21d9a5f2836"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "14fe49d38d45da0cbbb27b8db13b868d"
server_key_iv: "09ac378fcf49968e4edc887b91fa7b7b"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.709985 20885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:11.731324 20885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:11.733528 20885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:11.733690 20885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:11.733963 20885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:11.734292 20885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:11.734324 20885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.734378 20885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:11.734406 20885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.740186 20885 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:44853
I20260925 15:29:11.740235 21013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:44853 every 8 connection(s)
I20260925 15:29:11.740651 20885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.65@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:11.750418 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 20885
I20260925 15:29:11.750535 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:11.741386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:55529 (local address 127.1.163.126:41463)
0925 15:29:11.741468 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:11.741470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:11.741642 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:29:11.742110 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:11.742111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:11.742124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:11.742183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:11.743101 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.743620 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.744874 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.745078 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.745138 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:11.747387 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:11.747388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:11.747389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:11.747407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:11.748962 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.749425 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.749426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.749426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.749489 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.749794 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.749794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.749795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.749923 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:11.749995 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:11.750344 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 15:29:11.750442 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:29:11.750548 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:11.750746 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:11.741343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41463 (local address 127.1.163.65:55529)
0925 15:29:11.741507 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:11.741550 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:11.742024 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:11.742234 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:11.742238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:11.742811 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:11.743011 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:11.743016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.743712 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.743713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:11.744783 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:11.744787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.745174 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.745176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:11.745312 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:11.745833 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:11.745842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:11.747279 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:11.749086 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:11.749095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:11.749097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:11.749338 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:11.749573 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:11.749574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:11.749574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:11.749713 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:11.750000 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:11.750007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:11.750173 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 15:29:11.750320 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:29:11.750493 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260925 15:29:11.750804 1677 external_mini_cluster.cc:1428] Setting key 3ed463f9a76ff026919851a79b11aca7
I20260925 15:29:11.751555 21014 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41463
I20260925 15:29:11.751678 21014 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:11.751863 21014 heartbeater.cc:511] Master 127.1.163.126:41463 requested a full tablet report, sending...
I20260925 15:29:11.752501 20823 ts_manager.cc:194] Registered new tserver with Master: b05d7d51a68d4274bb2ac21d9a5f2836 (127.1.163.65:44853)
I20260925 15:29:11.753419 20823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.65@KRBTEST.COM'} at 127.1.163.65:55529
WARNING: no policy specified for kudu/127.1.163.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
Entry for principal kudu/127.1.163.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab.
I20260925 15:29:11.779043 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:11.862534 21019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:11.862799 21019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:11.862854 21019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:11.864086 21019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:11.864161 21019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:11.864204 21019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:11.865768 21019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.66.keytab
--principal=kudu/127.1.163.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:11.866072 21019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:11.866325 21019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:29:11.872052 21019 server_base.cc:1034] running on GCE node
W20260925 15:29:11.872030 21024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:11.872025 21027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:11.872012 21025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:11.872632 21019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:11.872922 21019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:11.874069 21019 hybrid_clock.cc:648] HybridClock initialized: now 1790350151874056 us; error 31 us; skew 500 ppm
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:11.876772 21019 init.cc:377] Logged in from keytab as kudu/127.1.163.66@KRBTEST.COM (short username kudu)
I20260925 15:29:11.877267 21019 webserver.cc:460] Webserver started at http://127.1.163.66:33775/ using document root <none> and password file <none>
I20260925 15:29:11.877506 21019 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:11.877553 21019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:11.877714 21019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:11.878824 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "eb7d8748683c440e80fb120bb518067b"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "3f91ba800128538e0cfb20c1403e3bc6"
server_key_iv: "e0eac4b86091b11a4629a27c93894905"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.879236 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "eb7d8748683c440e80fb120bb518067b"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "3f91ba800128538e0cfb20c1403e3bc6"
server_key_iv: "e0eac4b86091b11a4629a27c93894905"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.880825 21019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:11.881681 21034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.881958 21019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:11.882069 21019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "eb7d8748683c440e80fb120bb518067b"
format_stamp: "Formatted at 2026-09-25 15:29:11 on dist-test-slave-hvv5"
server_key: "3f91ba800128538e0cfb20c1403e3bc6"
server_key_iv: "e0eac4b86091b11a4629a27c93894905"
server_key_version: "encryptionkey@0"
I20260925 15:29:11.882153 21019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:11.900717 21019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:11.903188 21019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:11.903326 21019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:11.903643 21019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:11.903985 21019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:11.904044 21019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.904088 21019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:11.904115 21019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:11.910624 21019 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:35131
I20260925 15:29:11.910667 21147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:35131 every 8 connection(s)
I20260925 15:29:11.911085 21019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:29:11.914585 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21019
I20260925 15:29:11.914678 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:11.914944 1677 external_mini_cluster.cc:1428] Setting key 15bb90aa2b0279a426d10aeb6a1411ec
Sep 25 15:29:11 dist-test-slave-hvv5 krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350151, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.66@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:11.922539 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:11.912038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:40839 (local address 127.1.163.126:41463)
0925 15:29:11.912110 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:11.912113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:11.912327 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:29:11.912995 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:11.912997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:11.913008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:11.913131 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:11.914212 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.914798 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.916185 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.916374 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.916437 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:11.919156 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:11.919157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:11.919158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:11.919166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:11.920881 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.921441 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.921442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.921442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.921497 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:11.921822 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:11.921822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:11.921823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:11.921954 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:11.922005 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:11.922385 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:29:11.922469 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:29:11.922838 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:11.912009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41463 (local address 127.1.163.66:40839)
0925 15:29:11.912170 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:11.912213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:11.912853 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:11.913219 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:11.913223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:11.913829 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:11.914113 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:11.914120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.914898 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.914900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:11.916102 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:11.916105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:11.916462 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:11.916463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:11.916676 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:11.917247 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:11.917259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:11.919046 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:11.920972 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:11.920981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:11.920983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:11.921335 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:11.921573 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:11.921574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:11.921574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:11.921734 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:11.922026 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:11.922033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:11.922231 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:29:11.922366 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:29:11.922566 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 15:29:11.923372 21148 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41463
I20260925 15:29:11.923477 21148 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:11.923686 21148 heartbeater.cc:511] Master 127.1.163.126:41463 requested a full tablet report, sending...
I20260925 15:29:11.924129 20823 ts_manager.cc:194] Registered new tserver with Master: eb7d8748683c440e80fb120bb518067b (127.1.163.66:35131)
I20260925 15:29:11.924723 20823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.66@KRBTEST.COM'} at 127.1.163.66:40839
WARNING: no policy specified for kudu/127.1.163.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
Entry for principal kudu/127.1.163.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab.
I20260925 15:29:11.944408 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:12.028445 21153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:12.028797 21153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:12.028841 21153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:12.030059 21153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:12.030138 21153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:12.030182 21153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:12.031870 21153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.67.keytab
--principal=kudu/127.1.163.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:12.032167 21153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:12.032425 21153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:12.037856 21158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.037829 21161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.037988 21159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.038064 21153 server_base.cc:1034] running on GCE node
I20260925 15:29:12.038296 21153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:12.038508 21153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:12.039644 21153 hybrid_clock.cc:648] HybridClock initialized: now 1790350152039620 us; error 39 us; skew 500 ppm
Sep 25 15:29:12 dist-test-slave-hvv5 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350152, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:12.042332 21153 init.cc:377] Logged in from keytab as kudu/127.1.163.67@KRBTEST.COM (short username kudu)
I20260925 15:29:12.042807 21153 webserver.cc:460] Webserver started at http://127.1.163.67:45981/ using document root <none> and password file <none>
I20260925 15:29:12.043048 21153 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:12.043118 21153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:12.043272 21153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:12.044337 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "5e1d13d4120540adbbfc12ffd2a5eddf"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "f7725496270d8770ce19e16d478ddacc"
server_key_iv: "023e943f88986444e5062346051315fe"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.044731 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "5e1d13d4120540adbbfc12ffd2a5eddf"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "f7725496270d8770ce19e16d478ddacc"
server_key_iv: "023e943f88986444e5062346051315fe"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.046077 21153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:12.046897 21168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.047101 21153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:12.047200 21153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "5e1d13d4120540adbbfc12ffd2a5eddf"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "f7725496270d8770ce19e16d478ddacc"
server_key_iv: "023e943f88986444e5062346051315fe"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.047286 21153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:12.091142 21153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:12.093861 21153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:12.094059 21153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:12.094338 21153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:12.094694 21153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:12.094753 21153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.094816 21153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:12.094847 21153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.101625 21153 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:45231
I20260925 15:29:12.101668 21281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:45231 every 8 connection(s)
I20260925 15:29:12.102035 21153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:29:12 dist-test-slave-hvv5 krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350152, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.67@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:12.110176 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21153
I20260925 15:29:12.110280 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:12.110486 1677 external_mini_cluster.cc:1428] Setting key dd587ebc0d27ad5ae433cb476da7f0e6
I20260925 15:29:12.111714 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.102856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:47671 (local address 127.1.163.126:41463)
0925 15:29:12.102941 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.102944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.103110 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:29:12.103621 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.103622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.103633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.103692 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.104726 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.105265 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.106292 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.106496 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.106554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.108697 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.108699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.108699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:12.108706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.110065 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:12.110593 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:12.110594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:12.110594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:12.110662 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:12.111039 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:12.111040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:12.111040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:12.111164 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.111243 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.111531 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.111622 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:29:12.111918 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.102806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:41463 (local address 127.1.163.67:47671)
0925 15:29:12.102982 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.103018 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.103505 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.103761 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.103764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.104359 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:12.104600 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.104605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.105350 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.105351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.106208 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.106211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.106575 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.106576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.106682 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.107160 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:12.107168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:12.108608 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:12.110172 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:12.110180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:12.110183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:12.110501 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:12.110742 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:12.110743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:12.110744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:12.110948 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:12.111233 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:12.111240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:12.111393 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.111509 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.111678 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260925 15:29:12.112324 21282 heartbeater.cc:347] Connected to a master server at 127.1.163.126:41463
I20260925 15:29:12.112462 21282 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:12.112708 21282 heartbeater.cc:511] Master 127.1.163.126:41463 requested a full tablet report, sending...
I20260925 15:29:12.113076 20823 ts_manager.cc:194] Registered new tserver with Master: 5e1d13d4120540adbbfc12ffd2a5eddf (127.1.163.67:45231)
I20260925 15:29:12.113426 20823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.163.67@KRBTEST.COM'} at 127.1.163.67:47671
I20260925 15:29:12.122529 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:12.130759 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.124051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.1.163.126:41463)
0925 15:29:12.124153 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.124155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.124278 (+ 123us) server_negotiation.cc:407] Connection header received
0925 15:29:12.124367 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.124368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.124378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.124431 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.125122 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.125636 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.126399 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.126605 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.126693 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.127960 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.127961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.127962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:12.127969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.129405 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:12.129812 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:12.129813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:12.129814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:12.129878 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:12.130118 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:12.130119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:12.130119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:12.130284 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.130392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.130558 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.130649 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:29:12.132404 20823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:12.134248 20823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:12.141399 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.136970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:35131 (local address 127.0.0.1:41898)
0925 15:29:12.137238 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.137245 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.137286 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.137949 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.137950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.137952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.138007 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.138009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.138868 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.138869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.140047 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.140049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.141076 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.141077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.141155 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.141169 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.141246 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.141310 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260925 15:29:12.141438 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.137548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.1.163.66:35131)
0925 15:29:12.137782 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.137787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.137797 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:12.137836 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.137838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.137854 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.137949 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.138064 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.138767 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.140147 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.140977 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.141064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.141124 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.141277 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.141325 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260925 15:29:12.141597 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.137228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.1.163.65:44853)
0925 15:29:12.137417 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.137423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.137436 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:29:12.137483 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.137485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.137504 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.137736 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.138027 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.139236 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.140471 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.141063 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.141135 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.141196 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.141449 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.141502 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260925 15:29:12.141697 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.136970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:45231 (local address 127.0.0.1:43910)
0925 15:29:12.137111 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.137121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.137160 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.137942 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.137943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.137945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.137990 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.137991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.139361 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.139362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.140437 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.140441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.141460 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.141461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.141532 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.141537 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.141575 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.141618 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260925 15:29:12.141863 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.136912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:44853 (local address 127.0.0.1:46990)
0925 15:29:12.137020 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.137060 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.137132 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.137840 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.137843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.137847 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.137931 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.137933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.139347 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.139348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.140369 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.140371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.141231 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.141232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.141299 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.141301 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.141345 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.141379 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":68,"thread_start_us":58,"threads_started":1}
I20260925 15:29:12.142392 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.137543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.1.163.67:45231)
0925 15:29:12.137755 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.137760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.137771 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:12.137813 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.137815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.137831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.138075 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.138151 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.139214 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.140590 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.141349 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.142177 (+ 828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.142254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.142287 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.142327 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260925 15:29:12.142280 21082 tablet_service.cc:1467] Processing CreateTablet for tablet e2edf1b2371b45929d6c244926fe092b (DEFAULT_TABLE table=test-table [id=ff5f644dbfbb47b794c759512f5ff59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.142514 20948 tablet_service.cc:1467] Processing CreateTablet for tablet e2edf1b2371b45929d6c244926fe092b (DEFAULT_TABLE table=test-table [id=ff5f644dbfbb47b794c759512f5ff59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.142674 21082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2edf1b2371b45929d6c244926fe092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.142819 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2edf1b2371b45929d6c244926fe092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.143065 21216 tablet_service.cc:1467] Processing CreateTablet for tablet e2edf1b2371b45929d6c244926fe092b (DEFAULT_TABLE table=test-table [id=ff5f644dbfbb47b794c759512f5ff59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.143392 21216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2edf1b2371b45929d6c244926fe092b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.146380 21299 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Bootstrap starting.
I20260925 15:29:12.146584 21301 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Bootstrap starting.
I20260925 15:29:12.147032 21299 tablet_bootstrap.cc:654] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.147106 21301 tablet_bootstrap.cc:654] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.147524 21301 log.cc:824] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.147552 21299 log.cc:824] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.147912 21300 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: Bootstrap starting.
I20260925 15:29:12.148350 21301 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: No bootstrap required, opened a new log
I20260925 15:29:12.148360 21299 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: No bootstrap required, opened a new log
I20260925 15:29:12.148427 21299 ts_tablet_manager.cc:1397] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:29:12.148427 21301 ts_tablet_manager.cc:1397] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:12.148509 21300 tablet_bootstrap.cc:654] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.149021 21300 log.cc:824] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.149824 21300 tablet_bootstrap.cc:492] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: No bootstrap required, opened a new log
I20260925 15:29:12.149892 21300 ts_tablet_manager.cc:1397] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:29:12.150594 21301 raft_consensus.cc:348] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.150594 21299 raft_consensus.cc:348] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.150811 21301 raft_consensus.cc:374] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.150852 21301 raft_consensus.cc:729] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1d13d4120540adbbfc12ffd2a5eddf, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.150810 21299 raft_consensus.cc:374] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.150916 21299 raft_consensus.cc:729] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7d8748683c440e80fb120bb518067b, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.151006 21301 consensus_queue.cc:260] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.151055 21299 consensus_queue.cc:260] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.151304 21301 ts_tablet_manager.cc:1428] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:12.151325 21299 ts_tablet_manager.cc:1428] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:29:12.151360 21148 heartbeater.cc:503] Master 127.1.163.126:41463 was elected leader, sending a full tablet report...
I20260925 15:29:12.151438 21282 heartbeater.cc:503] Master 127.1.163.126:41463 was elected leader, sending a full tablet report...
I20260925 15:29:12.152037 21300 raft_consensus.cc:348] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.152207 21300 raft_consensus.cc:374] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.152256 21300 raft_consensus.cc:729] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05d7d51a68d4274bb2ac21d9a5f2836, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.152388 21300 consensus_queue.cc:260] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.152716 21300 ts_tablet_manager.cc:1428] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:29:12.154165 21014 heartbeater.cc:503] Master 127.1.163.126:41463 was elected leader, sending a full tablet report...
W20260925 15:29:12.161744 21149 tablet.cc:2369] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:12.210371 21305 raft_consensus.cc:482] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:12.210520 21305 raft_consensus.cc:504] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.210877 21305 leader_election.cc:288] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b05d7d51a68d4274bb2ac21d9a5f2836 (127.1.163.65:44853), eb7d8748683c440e80fb120bb518067b (127.1.163.66:35131)
I20260925 15:29:12.214982 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.211221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:36667 (local address 127.1.163.66:35131)
0925 15:29:12.211264 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.211266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.211383 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:29:12.211446 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.211449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.211461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.211505 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.211656 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.212353 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.213968 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.214571 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.214653 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.214662 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.214851 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.214914 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:29:12.214982 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.211085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:44853 (local address 127.1.163.67:49631)
0925 15:29:12.211169 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.211184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.211235 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.211405 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.211406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.211407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.211512 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.211514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.212470 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.212470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.213820 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.213823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.214672 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.214673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.214821 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.214824 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.214877 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.214907 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":123}
I20260925 15:29:12.214991 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.211123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:35131 (local address 127.1.163.67:36667)
0925 15:29:12.211320 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.211328 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.211393 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.211529 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.211530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.211532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.211590 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.211591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.212439 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.212440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.213820 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.213823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.214672 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.214673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.214766 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.214772 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.214837 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.214890 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":49,"thread_start_us":107,"threads_started":1}
I20260925 15:29:12.215044 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.211200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:49631 (local address 127.1.163.65:44853)
0925 15:29:12.211255 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.211257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.211287 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:29:12.211323 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.211324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.211332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.211399 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.211591 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.212414 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.213981 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.214571 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.214652 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.214662 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.214894 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.214968 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:29:12.215368 21102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2edf1b2371b45929d6c244926fe092b" candidate_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7d8748683c440e80fb120bb518067b" is_pre_election: true
I20260925 15:29:12.215409 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2edf1b2371b45929d6c244926fe092b" candidate_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" is_pre_election: true
I20260925 15:29:12.215533 21102 raft_consensus.cc:2454] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1d13d4120540adbbfc12ffd2a5eddf in term 0.
I20260925 15:29:12.215587 20968 raft_consensus.cc:2454] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1d13d4120540adbbfc12ffd2a5eddf in term 0.
I20260925 15:29:12.215786 21169 leader_election.cc:302] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1d13d4120540adbbfc12ffd2a5eddf, eb7d8748683c440e80fb120bb518067b; no voters:
I20260925 15:29:12.215963 21305 raft_consensus.cc:2781] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:12.216033 21305 raft_consensus.cc:482] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:12.216051 21305 raft_consensus.cc:3037] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.216843 21305 raft_consensus.cc:504] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.217080 21305 leader_election.cc:288] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [CANDIDATE]: Term 1 election: Requested vote from peers b05d7d51a68d4274bb2ac21d9a5f2836 (127.1.163.65:44853), eb7d8748683c440e80fb120bb518067b (127.1.163.66:35131)
I20260925 15:29:12.217226 21102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2edf1b2371b45929d6c244926fe092b" candidate_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7d8748683c440e80fb120bb518067b"
I20260925 15:29:12.217303 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2edf1b2371b45929d6c244926fe092b" candidate_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836"
I20260925 15:29:12.217341 21102 raft_consensus.cc:3037] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.217413 20968 raft_consensus.cc:3037] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.218231 20968 raft_consensus.cc:2454] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1d13d4120540adbbfc12ffd2a5eddf in term 1.
I20260925 15:29:12.218293 21102 raft_consensus.cc:2454] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1d13d4120540adbbfc12ffd2a5eddf in term 1.
I20260925 15:29:12.218480 21172 leader_election.cc:302] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1d13d4120540adbbfc12ffd2a5eddf, b05d7d51a68d4274bb2ac21d9a5f2836; no voters:
I20260925 15:29:12.218619 21305 raft_consensus.cc:2781] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:12.218853 21305 raft_consensus.cc:686] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 1 LEADER]: Becoming Leader. State: Replica: 5e1d13d4120540adbbfc12ffd2a5eddf, State: Running, Role: LEADER
I20260925 15:29:12.218984 21305 consensus_queue.cc:237] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } }
I20260925 15:29:12.219969 20822 catalog_manager.cc:5495] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1d13d4120540adbbfc12ffd2a5eddf (127.1.163.67). New cstate: current_term: 1 leader_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1d13d4120540adbbfc12ffd2a5eddf" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 45231 } health_report { overall_health: HEALTHY } } }
I20260925 15:29:12.236071 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.233570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.1.163.67:45231)
0925 15:29:12.233642 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.233644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.233652 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:12.233687 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.233688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.233705 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.233769 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:12.233959 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.234450 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.235222 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.235419 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.235496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.235688 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:12.235803 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:12.235906 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.235925 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.235946 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:29:12.238087 21102 raft_consensus.cc:1261] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 1 FOLLOWER]: Refusing update from remote peer 5e1d13d4120540adbbfc12ffd2a5eddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:12.238144 20968 raft_consensus.cc:1261] T e2edf1b2371b45929d6c244926fe092b P b05d7d51a68d4274bb2ac21d9a5f2836 [term 1 FOLLOWER]: Refusing update from remote peer 5e1d13d4120540adbbfc12ffd2a5eddf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:12.238483 21305 consensus_queue.cc:1035] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b05d7d51a68d4274bb2ac21d9a5f2836" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:12.238631 21309 consensus_queue.cc:1035] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7d8748683c440e80fb120bb518067b" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:12.241073 21312 mvcc.cc:205] Tried to move back new op lower bound from 7333274223564238848 to 7333274223489847296. Current Snapshot: MvccSnapshot[applied={T|T < 7333274223564238848}]
I20260925 15:29:12.241334 21313 mvcc.cc:205] Tried to move back new op lower bound from 7333274223564238848 to 7333274223489847296. Current Snapshot: MvccSnapshot[applied={T|T < 7333274223564238848}]
I20260925 15:29:12.241894 21314 mvcc.cc:205] Tried to move back new op lower bound from 7333274223564238848 to 7333274223489847296. Current Snapshot: MvccSnapshot[applied={T|T < 7333274223564238848}]
I20260925 15:29:12.243638 20823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:12.243729 20823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:12.244693 20823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee165e0e246a4b62b9c3de7d80675549: Sending DeleteTablet for 3 replicas of tablet e2edf1b2371b45929d6c244926fe092b
I20260925 15:29:12.245121 21082 tablet_service.cc:1514] Processing DeleteTablet for tablet e2edf1b2371b45929d6c244926fe092b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:41898
I20260925 15:29:12.245110 20948 tablet_service.cc:1514] Processing DeleteTablet for tablet e2edf1b2371b45929d6c244926fe092b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:46990
I20260925 15:29:12.245339 21321 tablet_replica.cc:330] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: stopping tablet replica
I20260925 15:29:12.245383 21216 tablet_service.cc:1514] Processing DeleteTablet for tablet e2edf1b2371b45929d6c244926fe092b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='kudu', principal='kudu/127.1.163.126@KRBTEST.COM'} at 127.0.0.1:43910
I20260925 15:29:12.245446 21321 raft_consensus.cc:2229] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:12.245568 21321 raft_consensus.cc:2258] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:12.245625 21323 tablet_replica.cc:330] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: stopping tablet replica
I20260925 15:29:12.245750 21323 raft_consensus.cc:2229] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:12.245899 21323 raft_consensus.cc:2258] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:12.245991 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20885
I20260925 15:29:12.246112 21321 ts_tablet_manager.cc:1905] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:12.246331 21323 ts_tablet_manager.cc:1905] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:12.248383 21323 ts_tablet_manager.cc:1918] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:12.248495 21323 log.cc:1196] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/e2edf1b2371b45929d6c244926fe092b
I20260925 15:29:12.248477 21321 ts_tablet_manager.cc:1918] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:12.248602 21321 log.cc:1196] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/e2edf1b2371b45929d6c244926fe092b
I20260925 15:29:12.248875 21323 ts_tablet_manager.cc:1939] T e2edf1b2371b45929d6c244926fe092b P 5e1d13d4120540adbbfc12ffd2a5eddf: Deleting consensus metadata
I20260925 15:29:12.248921 21321 ts_tablet_manager.cc:1939] T e2edf1b2371b45929d6c244926fe092b P eb7d8748683c440e80fb120bb518067b: Deleting consensus metadata
I20260925 15:29:12.249460 20807 catalog_manager.cc:4841] TS eb7d8748683c440e80fb120bb518067b (127.1.163.66:35131): tablet e2edf1b2371b45929d6c244926fe092b (table test-table [id=ff5f644dbfbb47b794c759512f5ff59b]) successfully deleted
I20260925 15:29:12.249614 20808 catalog_manager.cc:4841] TS 5e1d13d4120540adbbfc12ffd2a5eddf (127.1.163.67:45231): tablet e2edf1b2371b45929d6c244926fe092b (table test-table [id=ff5f644dbfbb47b794c759512f5ff59b]) successfully deleted
W20260925 15:29:12.253400 20810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:44853 (error 108)
I20260925 15:29:12.253651 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21019
I20260925 15:29:12.254760 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.254607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:44853 (local address 127.0.0.1:47004)
0925 15:29:12.254677 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.254718 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:29:12.254917 20810 catalog_manager.cc:4584] TS b05d7d51a68d4274bb2ac21d9a5f2836 (127.1.163.65:44853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2edf1b2371b45929d6c244926fe092b: Network error: Client connection negotiation failed: client connection to 127.1.163.65:44853: connect: Connection refused (error 111)
I20260925 15:29:12.258270 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21153
I20260925 15:29:12.262765 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 20791
2026-09-25T15:29:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:12Z Disabled control of system clock
I20260925 15:29:12.291545 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:40209
--webserver_interface=127.1.163.126
--webserver_port=0
--builtin_ntp_servers=127.1.163.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:12.374796 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:12.375034 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:12.375056 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:12.376360 21328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:12.376405 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:12.376418 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:12.376456 21328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:12.376482 21328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:12.377867 21328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:40209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:12.378146 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:12.378512 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:12.383683 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.383734 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.383709 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.383939 21328 server_base.cc:1034] running on GCE node
I20260925 15:29:12.384253 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:12.384601 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:12.385787 21328 hybrid_clock.cc:648] HybridClock initialized: now 1790350152385769 us; error 45 us; skew 500 ppm
I20260925 15:29:12.387049 21328 webserver.cc:460] Webserver started at http://127.1.163.126:35487/ using document root <none> and password file <none>
I20260925 15:29:12.387348 21328 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:12.387423 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:12.387584 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:12.388777 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "6bb2a18200684881a38ce90d87d9d0bd"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "dec602d01e25d4fb0991195f5366a07a"
server_key_iv: "accf7e5b54311d26db1cca83afed23ca"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.389145 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "6bb2a18200684881a38ce90d87d9d0bd"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "dec602d01e25d4fb0991195f5366a07a"
server_key_iv: "accf7e5b54311d26db1cca83afed23ca"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.390563 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:29:12.391350 21342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.391535 21328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:29:12.391634 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "6bb2a18200684881a38ce90d87d9d0bd"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "dec602d01e25d4fb0991195f5366a07a"
server_key_iv: "accf7e5b54311d26db1cca83afed23ca"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.391721 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:12.400101 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:12.400466 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:12.400657 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:12.404531 21328 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:40209
I20260925 15:29:12.404618 21394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:40209 every 8 connection(s)
I20260925 15:29:12.405005 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:12.406139 21395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.406435 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21328
I20260925 15:29:12.406543 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:12.406780 1677 external_mini_cluster.cc:1428] Setting key f4ec28fa340ffed123bb3375794c8a50
I20260925 15:29:12.409744 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: Bootstrap starting.
I20260925 15:29:12.410457 21395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.410766 21395 log.cc:824] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.411554 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: No bootstrap required, opened a new log
I20260925 15:29:12.412736 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.407403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.1.163.126:40209)
0925 15:29:12.407595 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.407602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.407627 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:29:12.408080 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.408088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.408154 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.408315 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.408530 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.409471 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.410552 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.411048 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.411161 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.411925 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.411929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.411929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:12.412138 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.412181 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:12.412195 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.412290 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.412388 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.412473 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
I20260925 15:29:12.414011 21395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } }
I20260925 15:29:12.414157 21395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.414181 21395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb2a18200684881a38ce90d87d9d0bd, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.414304 21395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } }
I20260925 15:29:12.414378 21395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:12.414402 21395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:12.414429 21395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.414989 21395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } }
I20260925 15:29:12.415076 21395 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb2a18200684881a38ce90d87d9d0bd; no voters:
I20260925 15:29:12.415196 21395 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:12.415274 21400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:12.415442 21400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [term 1 LEADER]: Becoming Leader. State: Replica: 6bb2a18200684881a38ce90d87d9d0bd, State: Running, Role: LEADER
I20260925 15:29:12.415491 21395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:12.415529 21400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } }
I20260925 15:29:12.415913 21402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb2a18200684881a38ce90d87d9d0bd. Latest consensus state: current_term: 1 leader_uuid: "6bb2a18200684881a38ce90d87d9d0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } } }
I20260925 15:29:12.416008 21402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:12.415884 21401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb2a18200684881a38ce90d87d9d0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb2a18200684881a38ce90d87d9d0bd" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 40209 } } }
I20260925 15:29:12.416160 21401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:12.416224 21407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:12.416920 21407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:12.418351 21407 catalog_manager.cc:1347] Generated new cluster ID: f2482c9187d6447cae819f8523fa0920
I20260925 15:29:12.418428 21407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:12.433498 21407 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:12.434273 21407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:12.437424 21407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: Generated new TSK 0
I20260925 15:29:12.437626 21407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:12.446095 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.65:0
--local_ip_for_outbound_sockets=127.1.163.65
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--builtin_ntp_servers=127.1.163.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:29:12.531657 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:12.532039 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:12.532119 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:12.533427 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:12.533493 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:12.533537 21419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.65
I20260925 15:29:12.534961 21419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.163.65
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.65
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:12.535231 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:12.535553 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:12.541551 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.541586 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.541551 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.542032 21419 server_base.cc:1034] running on GCE node
I20260925 15:29:12.542275 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:12.542537 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:12.543704 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790350152543690 us; error 26 us; skew 500 ppm
I20260925 15:29:12.545032 21419 webserver.cc:460] Webserver started at http://127.1.163.65:44929/ using document root <none> and password file <none>
I20260925 15:29:12.545284 21419 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:12.545390 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:12.545552 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:12.546662 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data/instance:
uuid: "bd065825f6034a22b86513ee7cc566b2"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "7b88dec491bad5747c16434e239fd76f"
server_key_iv: "f8cc449f8eb619004e121a010665397b"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.547039 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance:
uuid: "bd065825f6034a22b86513ee7cc566b2"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "7b88dec491bad5747c16434e239fd76f"
server_key_iv: "f8cc449f8eb619004e121a010665397b"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.548434 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:12.549288 21433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.549486 21419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:12.549587 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal
uuid: "bd065825f6034a22b86513ee7cc566b2"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "7b88dec491bad5747c16434e239fd76f"
server_key_iv: "f8cc449f8eb619004e121a010665397b"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.549675 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:12.560698 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:12.561025 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:12.561182 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:12.561429 21419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:12.561762 21419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:12.561816 21419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.561858 21419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:12.561888 21419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.567512 21419 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.65:34785
I20260925 15:29:12.567584 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.65:34785 every 8 connection(s)
I20260925 15:29:12.567927 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/data/info.pb
I20260925 15:29:12.571456 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21419
I20260925 15:29:12.571558 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-0/wal/instance
I20260925 15:29:12.571821 1677 external_mini_cluster.cc:1428] Setting key 51a2f4eebb90ff5e563c696409b5fd45
I20260925 15:29:12.572906 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.66:0
--local_ip_for_outbound_sockets=127.1.163.66
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--builtin_ntp_servers=127.1.163.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:12.573633 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.568669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.65:54347 (local address 127.1.163.126:40209)
0925 15:29:12.568730 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.568732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.569023 (+ 291us) server_negotiation.cc:407] Connection header received
0925 15:29:12.569548 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.569549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.569564 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.569621 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.570274 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.570828 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.571971 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.572248 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.572341 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.573359 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.573361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.573361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:12.573366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.573383 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:12.573388 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.573444 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.573524 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.573553 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:29:12.573863 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.568698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40209 (local address 127.1.163.65:54347)
0925 15:29:12.568885 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.568921 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.569428 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.569678 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.569681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.569689 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:12.570177 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.570181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.570922 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.570924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.571895 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.571899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.572359 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.572361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.572525 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.573203 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:12.573204 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:12.573238 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:12.573262 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:12.573447 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:12.573452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:12.573455 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.573517 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.573622 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260925 15:29:12.574465 21547 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40209
I20260925 15:29:12.574555 21547 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:12.574759 21547 heartbeater.cc:511] Master 127.1.163.126:40209 requested a full tablet report, sending...
I20260925 15:29:12.575345 21359 ts_manager.cc:194] Registered new tserver with Master: bd065825f6034a22b86513ee7cc566b2 (127.1.163.65:34785)
I20260925 15:29:12.576287 21359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.65:54347
W20260925 15:29:12.658056 21550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:12.658281 21550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:12.658304 21550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:12.659428 21550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:12.659464 21550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:12.659490 21550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.66
I20260925 15:29:12.660858 21550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.163.66
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.66
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:12.661079 21550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:12.661348 21550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:12.666592 21558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.666599 21555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.666760 21550 server_base.cc:1034] running on GCE node
W20260925 15:29:12.666996 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.667250 21550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:12.667518 21550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:12.668678 21550 hybrid_clock.cc:648] HybridClock initialized: now 1790350152668672 us; error 38 us; skew 500 ppm
I20260925 15:29:12.670110 21550 webserver.cc:460] Webserver started at http://127.1.163.66:36009/ using document root <none> and password file <none>
I20260925 15:29:12.670362 21550 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:12.670449 21550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:12.670603 21550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:12.671743 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data/instance:
uuid: "4d50e7030f284af49e79de82ac94c62c"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "161ed0429edd41366ed8148929c940e0"
server_key_iv: "a981dd26e0c551bdc82000816edaee61"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.672111 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance:
uuid: "4d50e7030f284af49e79de82ac94c62c"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "161ed0429edd41366ed8148929c940e0"
server_key_iv: "a981dd26e0c551bdc82000816edaee61"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.673553 21550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:12.674371 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.674577 21550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:12.674691 21550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal
uuid: "4d50e7030f284af49e79de82ac94c62c"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "161ed0429edd41366ed8148929c940e0"
server_key_iv: "a981dd26e0c551bdc82000816edaee61"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.674772 21550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:12.690754 21550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:12.691130 21550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:12.691285 21550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:12.691519 21550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:12.691847 21550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:12.691910 21550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.691957 21550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:12.691982 21550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.698295 21550 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.66:39875
I20260925 15:29:12.698392 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.66:39875 every 8 connection(s)
I20260925 15:29:12.698748 21550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/data/info.pb
I20260925 15:29:12.698916 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21550
I20260925 15:29:12.699052 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal/instance
I20260925 15:29:12.699254 1677 external_mini_cluster.cc:1428] Setting key 3c34fa68b4f76b1c44f23ea303e36aca
I20260925 15:29:12.700381 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.163.67:0
--local_ip_for_outbound_sockets=127.1.163.67
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--builtin_ntp_servers=127.1.163.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:29:12.705343 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.699562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:34027 (local address 127.1.163.126:40209)
0925 15:29:12.699617 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.699619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.699939 (+ 320us) server_negotiation.cc:407] Connection header received
0925 15:29:12.700726 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.700727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.700738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.700794 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.701723 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.702529 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.703639 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.703867 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.703951 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.705020 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.705022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.705022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:12.705028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.705044 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:12.705048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.705119 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.705221 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.705252 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:29:12.705579 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.699581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40209 (local address 127.1.163.66:34027)
0925 15:29:12.699781 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.699817 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.700592 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.700914 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.700918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.700929 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:12.701631 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.701638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.702657 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.702658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.703510 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.703512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.703960 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.703962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.704109 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.704819 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:12.704820 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:12.704849 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:12.704875 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:12.705133 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:12.705138 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:12.705141 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.705235 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.705357 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260925 15:29:12.706025 21678 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40209
I20260925 15:29:12.706144 21678 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:12.706375 21678 heartbeater.cc:511] Master 127.1.163.126:40209 requested a full tablet report, sending...
I20260925 15:29:12.706911 21359 ts_manager.cc:194] Registered new tserver with Master: 4d50e7030f284af49e79de82ac94c62c (127.1.163.66:39875)
I20260925 15:29:12.707324 21359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.66:34027
W20260925 15:29:12.784663 21681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:12.784895 21681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:12.784931 21681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:12.786239 21681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:12.786307 21681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:12.786352 21681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.163.67
I20260925 15:29:12.787806 21681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.163.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.163.67
--webserver_port=0
--tserver_master_addrs=127.1.163.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.163.67
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:12.788096 21681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:12.788414 21681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:12.793730 21687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:12.793725 21686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.793802 21681 server_base.cc:1034] running on GCE node
W20260925 15:29:12.793726 21689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:12.794169 21681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:12.794427 21681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:12.795511 21681 hybrid_clock.cc:648] HybridClock initialized: now 1790350152795495 us; error 31 us; skew 500 ppm
I20260925 15:29:12.796868 21681 webserver.cc:460] Webserver started at http://127.1.163.67:46025/ using document root <none> and password file <none>
I20260925 15:29:12.797117 21681 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:12.797187 21681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:12.797336 21681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:12.798434 21681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data/instance:
uuid: "8b98a6b48d9d4e23b83a4c3706d0991a"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "4bc6d7782e010c289fa1ac60e50fc0d6"
server_key_iv: "12492998e5f1a5f5bc425de1408e08e0"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.798796 21681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance:
uuid: "8b98a6b48d9d4e23b83a4c3706d0991a"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "4bc6d7782e010c289fa1ac60e50fc0d6"
server_key_iv: "12492998e5f1a5f5bc425de1408e08e0"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.800136 21681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:12.800977 21695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.801149 21681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:29:12.801250 21681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal
uuid: "8b98a6b48d9d4e23b83a4c3706d0991a"
format_stamp: "Formatted at 2026-09-25 15:29:12 on dist-test-slave-hvv5"
server_key: "4bc6d7782e010c289fa1ac60e50fc0d6"
server_key_iv: "12492998e5f1a5f5bc425de1408e08e0"
server_key_version: "encryptionkey@0"
I20260925 15:29:12.801330 21681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:12.810225 21681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:12.810536 21681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:12.810683 21681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:12.810930 21681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:29:12.811256 21681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:29:12.811311 21681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.811345 21681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:29:12.811391 21681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:12.817332 21681 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.67:34325
I20260925 15:29:12.817404 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.67:34325 every 8 connection(s)
I20260925 15:29:12.817734 21681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/data/info.pb
I20260925 15:29:12.823055 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.818525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.67:42803 (local address 127.1.163.126:40209)
0925 15:29:12.818608 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.818611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.818794 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:29:12.819280 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.819281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.819292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.819357 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.820000 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.820501 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.821555 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.821745 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.821801 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.822829 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.822830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.822830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:12.822835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.822847 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:12.822850 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.822903 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.822971 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.822991 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:29:12.823259 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.818474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:40209 (local address 127.1.163.67:42803)
0925 15:29:12.818651 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.818687 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.819191 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.819412 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.819416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.819424 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:12.819917 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.819922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.820647 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.820648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.821473 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.821476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.821921 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.821922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.822052 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.822691 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:29:12.822692 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:12.822728 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:29:12.822752 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:12.822907 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:12.822909 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:12.822911 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.822959 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.823038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 15:29:12.823792 21809 heartbeater.cc:347] Connected to a master server at 127.1.163.126:40209
I20260925 15:29:12.823873 21809 heartbeater.cc:464] Registering TS with master...
I20260925 15:29:12.824018 21809 heartbeater.cc:511] Master 127.1.163.126:40209 requested a full tablet report, sending...
I20260925 15:29:12.824394 21359 ts_manager.cc:194] Registered new tserver with Master: 8b98a6b48d9d4e23b83a4c3706d0991a (127.1.163.67:34325)
I20260925 15:29:12.824805 21359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.163.67:42803
I20260925 15:29:12.826054 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21681
I20260925 15:29:12.826145 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal/instance
I20260925 15:29:12.826391 1677 external_mini_cluster.cc:1428] Setting key 61ecfd52042b2602b58b864acf25eafc
I20260925 15:29:12.827560 1677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:29:12.831950 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.828945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.1.163.126:40209)
0925 15:29:12.829004 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.829007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.829131 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:29:12.829208 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.829209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.829220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.829291 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:12.829475 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.829946 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.830699 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.830891 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.830979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.831704 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:12.831705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:12.831706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:12.831710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:12.831724 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:12.831728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:12.831781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.831843 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.831866 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:29:12.833705 21359 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:29:12.835263 21359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:29:12.841642 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.66:39875 (local address 127.0.0.1:47132)
0925 15:29:12.837993 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.838005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.838053 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.838383 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.838383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.838385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.838432 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.838433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839316 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.839317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.840566 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.840569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841316 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.841397 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841411 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.841501 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.841557 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260925 15:29:12.841737 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.1.163.65:34785)
0925 15:29:12.837925 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.837930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.837941 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:12.837983 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.837985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.838001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.838292 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.838358 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839460 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.840608 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841354 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841568 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.841601 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.841639 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":112,"threads_started":1}
I20260925 15:29:12.841977 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.1.163.67:34325)
0925 15:29:12.837829 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.837835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.837997 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:29:12.838039 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.838041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.838055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.838122 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.838292 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839438 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.840739 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841436 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841495 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.841874 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.841921 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260925 15:29:12.842003 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.1.163.66:39875)
0925 15:29:12.838155 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.838159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.838169 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:12.838208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.838210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.838226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.838300 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.838528 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839204 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.840678 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841202 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841295 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841393 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.841869 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.841917 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260925 15:29:12.841723 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:34785 (local address 127.0.0.1:57756)
0925 15:29:12.837658 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.837697 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.837760 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.838137 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.838140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.838142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.838209 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.838211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839561 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.839562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.840486 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.840489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841490 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.841547 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841549 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.841591 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.841621 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260925 15:29:12.842187 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.837474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34325 (local address 127.0.0.1:34426)
0925 15:29:12.837862 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.837874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.837938 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.838185 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.838186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.838188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.838215 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.838216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.839528 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.839529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.840659 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.840661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.841604 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.841606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.841669 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.841674 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.841725 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.841772 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260925 15:29:12.842847 21481 tablet_service.cc:1467] Processing CreateTablet for tablet 07896e04dd4846239b79bca34705e7d4 (DEFAULT_TABLE table=test-table [id=915961218d894fa395dbfb863a5e7b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.843058 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 07896e04dd4846239b79bca34705e7d4 (DEFAULT_TABLE table=test-table [id=915961218d894fa395dbfb863a5e7b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.843221 21481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896e04dd4846239b79bca34705e7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.843199 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 07896e04dd4846239b79bca34705e7d4 (DEFAULT_TABLE table=test-table [id=915961218d894fa395dbfb863a5e7b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:29:12.843429 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896e04dd4846239b79bca34705e7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.843499 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07896e04dd4846239b79bca34705e7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:12.846122 21823 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Bootstrap starting.
I20260925 15:29:12.846784 21824 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Bootstrap starting.
I20260925 15:29:12.846861 21823 tablet_bootstrap.cc:654] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.847191 21823 log.cc:824] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.847434 21824 tablet_bootstrap.cc:654] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.847761 21824 log.cc:824] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.847997 21823 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: No bootstrap required, opened a new log
I20260925 15:29:12.848075 21823 ts_tablet_manager.cc:1397] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:12.848107 21825 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Bootstrap starting.
I20260925 15:29:12.848445 21824 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: No bootstrap required, opened a new log
I20260925 15:29:12.848529 21824 ts_tablet_manager.cc:1397] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:12.848771 21825 tablet_bootstrap.cc:654] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:12.849112 21825 log.cc:824] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:12.849954 21825 tablet_bootstrap.cc:492] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: No bootstrap required, opened a new log
I20260925 15:29:12.850026 21825 ts_tablet_manager.cc:1397] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:29:12.850744 21824 raft_consensus.cc:348] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.850917 21824 raft_consensus.cc:374] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.850962 21824 raft_consensus.cc:729] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d50e7030f284af49e79de82ac94c62c, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.851083 21824 consensus_queue.cc:260] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.851155 21823 raft_consensus.cc:348] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.851320 21823 raft_consensus.cc:374] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.851351 21824 ts_tablet_manager.cc:1428] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:29:12.851367 21823 raft_consensus.cc:729] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd065825f6034a22b86513ee7cc566b2, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.851485 21678 heartbeater.cc:503] Master 127.1.163.126:40209 was elected leader, sending a full tablet report...
I20260925 15:29:12.851481 21823 consensus_queue.cc:260] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.851805 21547 heartbeater.cc:503] Master 127.1.163.126:40209 was elected leader, sending a full tablet report...
I20260925 15:29:12.852020 21823 ts_tablet_manager.cc:1428] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:29:12.852859 21825 raft_consensus.cc:348] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.853042 21825 raft_consensus.cc:374] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:12.853098 21825 raft_consensus.cc:729] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b98a6b48d9d4e23b83a4c3706d0991a, State: Initialized, Role: FOLLOWER
I20260925 15:29:12.853209 21825 consensus_queue.cc:260] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.853482 21825 ts_tablet_manager.cc:1428] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:29:12.853502 21809 heartbeater.cc:503] Master 127.1.163.126:40209 was elected leader, sending a full tablet report...
I20260925 15:29:12.925370 21829 raft_consensus.cc:482] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:29:12.925560 21829 raft_consensus.cc:504] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.925982 21829 leader_election.cc:288] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd065825f6034a22b86513ee7cc566b2 (127.1.163.65:34785), 8b98a6b48d9d4e23b83a4c3706d0991a (127.1.163.67:34325)
I20260925 15:29:12.929051 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.926250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:35341 (local address 127.1.163.67:34325)
0925 15:29:12.926361 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.926363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.926371 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:12.926411 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.926412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.926432 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.926503 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.926665 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.927170 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928273 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.928664 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.928764 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.928923 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.928960 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:29:12.929080 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.926275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.163.66:34029 (local address 127.1.163.65:34785)
0925 15:29:12.926323 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.926326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.926450 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:29:12.926523 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.926524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.926536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.926603 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:29:12.926812 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.927261 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928286 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.928715 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928821 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.928829 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.928968 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.929005 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:29:12.929095 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.926189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.67:34325 (local address 127.1.163.66:35341)
0925 15:29:12.926252 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.926266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.926312 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.926508 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.926509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.926511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.926590 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.926592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.927266 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.927268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.928179 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.928182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.928754 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.928827 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.928830 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.928907 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.928953 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:29:12.929324 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.926191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:34785 (local address 127.1.163.66:34029)
0925 15:29:12.926359 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.926370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:12.926466 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:12.926620 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:12.926620 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:12.926622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:29:12.926697 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.926698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.927345 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.927347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.928186 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:12.928188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.928819 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.928820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:12.928897 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.928901 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:29:12.928960 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:29:12.928998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":14,"thread_start_us":97,"threads_started":1}
I20260925 15:29:12.929459 21763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896e04dd4846239b79bca34705e7d4" candidate_uuid: "4d50e7030f284af49e79de82ac94c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" is_pre_election: true
I20260925 15:29:12.929663 21763 raft_consensus.cc:2454] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d50e7030f284af49e79de82ac94c62c in term 0.
I20260925 15:29:12.929675 21501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896e04dd4846239b79bca34705e7d4" candidate_uuid: "4d50e7030f284af49e79de82ac94c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd065825f6034a22b86513ee7cc566b2" is_pre_election: true
I20260925 15:29:12.929944 21501 raft_consensus.cc:2454] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d50e7030f284af49e79de82ac94c62c in term 0.
I20260925 15:29:12.930094 21565 leader_election.cc:302] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d50e7030f284af49e79de82ac94c62c, 8b98a6b48d9d4e23b83a4c3706d0991a; no voters:
I20260925 15:29:12.930286 21829 raft_consensus.cc:2781] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:29:12.930342 21829 raft_consensus.cc:482] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:29:12.930372 21829 raft_consensus.cc:3037] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.931092 21829 raft_consensus.cc:504] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.931250 21829 leader_election.cc:288] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [CANDIDATE]: Term 1 election: Requested vote from peers bd065825f6034a22b86513ee7cc566b2 (127.1.163.65:34785), 8b98a6b48d9d4e23b83a4c3706d0991a (127.1.163.67:34325)
I20260925 15:29:12.931391 21763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896e04dd4846239b79bca34705e7d4" candidate_uuid: "4d50e7030f284af49e79de82ac94c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a"
I20260925 15:29:12.931418 21501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07896e04dd4846239b79bca34705e7d4" candidate_uuid: "4d50e7030f284af49e79de82ac94c62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd065825f6034a22b86513ee7cc566b2"
I20260925 15:29:12.931495 21763 raft_consensus.cc:3037] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.931497 21501 raft_consensus.cc:3037] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:12.932222 21501 raft_consensus.cc:2454] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d50e7030f284af49e79de82ac94c62c in term 1.
I20260925 15:29:12.932266 21763 raft_consensus.cc:2454] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d50e7030f284af49e79de82ac94c62c in term 1.
I20260925 15:29:12.932420 21567 leader_election.cc:302] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d50e7030f284af49e79de82ac94c62c, bd065825f6034a22b86513ee7cc566b2; no voters:
I20260925 15:29:12.932571 21829 raft_consensus.cc:2781] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:12.932818 21829 raft_consensus.cc:686] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 1 LEADER]: Becoming Leader. State: Replica: 4d50e7030f284af49e79de82ac94c62c, State: Running, Role: LEADER
I20260925 15:29:12.932915 21829 consensus_queue.cc:237] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } }
I20260925 15:29:12.933794 21359 catalog_manager.cc:5495] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d50e7030f284af49e79de82ac94c62c (127.1.163.66). New cstate: current_term: 1 leader_uuid: "4d50e7030f284af49e79de82ac94c62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d50e7030f284af49e79de82ac94c62c" member_type: VOTER last_known_addr { host: "127.1.163.66" port: 39875 } health_report { overall_health: HEALTHY } } }
W20260925 15:29:12.949434 21679 tablet.cc:2369] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:12.957828 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.955196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.1.163.66:39875)
0925 15:29:12.955256 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:12.955258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:12.955267 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:12.955354 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:12.955354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:12.955381 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:12.955470 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:12.955611 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.956120 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.957000 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:12.957216 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:12.957303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:12.957470 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:12.957581 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:12.957678 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:12.957704 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:29:12.957725 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:29:12.959805 21501 raft_consensus.cc:1261] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 1 FOLLOWER]: Refusing update from remote peer 4d50e7030f284af49e79de82ac94c62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:12.959827 21763 raft_consensus.cc:1261] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 1 FOLLOWER]: Refusing update from remote peer 4d50e7030f284af49e79de82ac94c62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:12.960201 21833 consensus_queue.cc:1035] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd065825f6034a22b86513ee7cc566b2" member_type: VOTER last_known_addr { host: "127.1.163.65" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:12.960345 21829 consensus_queue.cc:1035] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b98a6b48d9d4e23b83a4c3706d0991a" member_type: VOTER last_known_addr { host: "127.1.163.67" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:12.963030 21837 mvcc.cc:205] Tried to move back new op lower bound from 7333274226520604672 to 7333274226413899776. Current Snapshot: MvccSnapshot[applied={T|T < 7333274226520604672}]
I20260925 15:29:12.963106 21838 mvcc.cc:205] Tried to move back new op lower bound from 7333274226520604672 to 7333274226413899776. Current Snapshot: MvccSnapshot[applied={T|T < 7333274226520604672}]
I20260925 15:29:12.963248 21836 mvcc.cc:205] Tried to move back new op lower bound from 7333274226520604672 to 7333274226413899776. Current Snapshot: MvccSnapshot[applied={T|T < 7333274226520604672}]
I20260925 15:29:12.965963 21359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:12.966074 21359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:29:12.966879 21359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bb2a18200684881a38ce90d87d9d0bd: Sending DeleteTablet for 3 replicas of tablet 07896e04dd4846239b79bca34705e7d4
I20260925 15:29:12.967258 21481 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896e04dd4846239b79bca34705e7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='slave'} at 127.0.0.1:57756
I20260925 15:29:12.967375 21612 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896e04dd4846239b79bca34705e7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='slave'} at 127.0.0.1:47132
I20260925 15:29:12.967514 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 07896e04dd4846239b79bca34705e7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:29:12 UTC) from {username='slave'} at 127.0.0.1:34426
I20260925 15:29:12.967569 21846 tablet_replica.cc:330] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: stopping tablet replica
I20260925 15:29:12.967686 21846 raft_consensus.cc:2229] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:12.967800 21846 raft_consensus.cc:2258] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:12.967984 21848 tablet_replica.cc:330] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: stopping tablet replica
I20260925 15:29:12.968047 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21419
I20260925 15:29:12.968084 21848 raft_consensus.cc:2229] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:29:12.968186 21848 raft_consensus.cc:2258] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:12.968304 21846 ts_tablet_manager.cc:1905] T 07896e04dd4846239b79bca34705e7d4 P bd065825f6034a22b86513ee7cc566b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:12.968425 21847 tablet_replica.cc:330] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: stopping tablet replica
I20260925 15:29:12.968508 21847 raft_consensus.cc:2229] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:29:12.968644 21847 raft_consensus.cc:2258] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:29:12.969040 21847 ts_tablet_manager.cc:1905] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:12.970811 21847 ts_tablet_manager.cc:1918] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:12.970868 21847 log.cc:1196] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-1/wal/wals/07896e04dd4846239b79bca34705e7d4
I20260925 15:29:12.971064 21847 ts_tablet_manager.cc:1939] T 07896e04dd4846239b79bca34705e7d4 P 4d50e7030f284af49e79de82ac94c62c: Deleting consensus metadata
I20260925 15:29:12.971450 21343 catalog_manager.cc:4841] TS 4d50e7030f284af49e79de82ac94c62c (127.1.163.66:39875): tablet 07896e04dd4846239b79bca34705e7d4 (table test-table [id=915961218d894fa395dbfb863a5e7b53]) successfully deleted
W20260925 15:29:12.975860 21345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.163.65:34785 (error 108)
I20260925 15:29:12.976140 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21550
I20260925 15:29:12.976392 21848 ts_tablet_manager.cc:1905] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:29:12.976496 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:12.976352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.65:34785 (local address 127.0.0.1:57768)
0925 15:29:12.976410 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:12.976455 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.65:34785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:29:12.976639 21345 catalog_manager.cc:4584] TS bd065825f6034a22b86513ee7cc566b2 (127.1.163.65:34785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07896e04dd4846239b79bca34705e7d4: Network error: Client connection negotiation failed: client connection to 127.1.163.65:34785: connect: Connection refused (error 111)
I20260925 15:29:12.978667 21848 ts_tablet_manager.cc:1918] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:29:12.978758 21848 log.cc:1196] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Deleting WAL directory at /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349904034090-1677-0/minicluster-data/ts-2/wal/wals/07896e04dd4846239b79bca34705e7d4
I20260925 15:29:12.979027 21848 ts_tablet_manager.cc:1939] T 07896e04dd4846239b79bca34705e7d4 P 8b98a6b48d9d4e23b83a4c3706d0991a: Deleting consensus metadata
I20260925 15:29:12.979844 21343 catalog_manager.cc:4841] TS 8b98a6b48d9d4e23b83a4c3706d0991a (127.1.163.67:34325): tablet 07896e04dd4846239b79bca34705e7d4 (table test-table [id=915961218d894fa395dbfb863a5e7b53]) successfully deleted
I20260925 15:29:12.982112 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21681
I20260925 15:29:12.986742 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21328
2026-09-25T15:29:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (723 ms)
[----------] 6 tests from EncryptionPolicyTest (8834 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:13 dist-test-slave-hvv5 krb5kdc[21852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:13 dist-test-slave-hvv5 krb5kdc[21852](info): set up 2 sockets
Sep 25 15:29:13 dist-test-slave-hvv5 krb5kdc[21852](info): commencing operation
krb5kdc: starting...
W20260925 15:29:14.060878 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:14.199416 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:14.281980 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:14.282209 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:14.282234 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:14.283367 21865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:14.283416 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:14.283432 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:14.283444 21865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:14.283456 21865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:14.284931 21865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:14.285144 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:14.285323 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:14.290419 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:14.290570 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:14.290572 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:14.290659 21865 server_base.cc:1034] running on GCE node
I20260925 15:29:14.291209 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:14.291541 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:14.292716 21865 hybrid_clock.cc:648] HybridClock initialized: now 1790350154292672 us; error 63 us; skew 500 ppm
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:14.302399 21865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:14.302932 21865 webserver.cc:460] Webserver started at http://127.0.0.1:34691/ using document root <none> and password file <none>
I20260925 15:29:14.303184 21865 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:14.303232 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:14.303390 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:14.304528 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "656c402bac05413f8be78350d945d35b"
format_stamp: "Formatted at 2026-09-25 15:29:14 on dist-test-slave-hvv5"
server_key: "c9701a5cc6fcbdfeba7dfe61f255ebac"
server_key_iv: "f145926ac70ef005fbd5ad9be8d238ec"
server_key_version: "encryptionkey@0"
I20260925 15:29:14.305001 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "656c402bac05413f8be78350d945d35b"
format_stamp: "Formatted at 2026-09-25 15:29:14 on dist-test-slave-hvv5"
server_key: "c9701a5cc6fcbdfeba7dfe61f255ebac"
server_key_iv: "f145926ac70ef005fbd5ad9be8d238ec"
server_key_version: "encryptionkey@0"
I20260925 15:29:14.306511 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:14.307436 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:14.307688 21865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:14.307821 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "656c402bac05413f8be78350d945d35b"
format_stamp: "Formatted at 2026-09-25 15:29:14 on dist-test-slave-hvv5"
server_key: "c9701a5cc6fcbdfeba7dfe61f255ebac"
server_key_iv: "f145926ac70ef005fbd5ad9be8d238ec"
server_key_version: "encryptionkey@0"
I20260925 15:29:14.307924 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:14.338819 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:14.341351 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:14.341531 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:14.346064 21865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36875
I20260925 15:29:14.346112 21932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36875 every 8 connection(s)
I20260925 15:29:14.346513 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:14.347525 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:14.349933 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b: Bootstrap starting.
I20260925 15:29:14.350481 21933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:14.350720 21933 log.cc:824] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:14.351457 21933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b: No bootstrap required, opened a new log
I20260925 15:29:14.353132 21933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } }
I20260925 15:29:14.353286 21933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:14.353309 21933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656c402bac05413f8be78350d945d35b, State: Initialized, Role: FOLLOWER
I20260925 15:29:14.353459 21933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } }
I20260925 15:29:14.353541 21933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:14.353566 21933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:14.353595 21933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:14.354111 21933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } }
I20260925 15:29:14.354197 21933 leader_election.cc:302] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656c402bac05413f8be78350d945d35b; no voters:
I20260925 15:29:14.354347 21933 leader_election.cc:288] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:14.354485 21936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:14.354600 21933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:14.354668 21936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [term 1 LEADER]: Becoming Leader. State: Replica: 656c402bac05413f8be78350d945d35b, State: Running, Role: LEADER
I20260925 15:29:14.354764 21936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } }
I20260925 15:29:14.355255 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656c402bac05413f8be78350d945d35b. Latest consensus state: current_term: 1 leader_uuid: "656c402bac05413f8be78350d945d35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } } }
I20260925 15:29:14.355347 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:14.355368 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656c402bac05413f8be78350d945d35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656c402bac05413f8be78350d945d35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36875 } } }
I20260925 15:29:14.355460 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:14.355620 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21865
I20260925 15:29:14.355629 21945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:14.355727 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:14.355947 1677 external_mini_cluster.cc:1428] Setting key e35a3076ecd697d49057d44bd87fc186
I20260925 15:29:14.356362 21945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:14.358279 21945 catalog_manager.cc:1347] Generated new cluster ID: 3f20ebd565ec45f0ae3b06a9b19ef052
I20260925 15:29:14.358356 21945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:29:14.368641 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:14.356456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50849 (local address 127.0.0.1:36875)
0925 15:29:14.356691 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:14.356701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:14.356744 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:29:14.356870 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:14.356879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:14.356939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:14.357640 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:14.357926 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:14.359314 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:14.360381 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:14.361068 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:14.361184 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:14.364112 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:14.364117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:14.364118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:14.364132 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:14.366298 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:14.366811 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:14.366814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:14.366815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:14.366953 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:14.367297 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:14.367299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:14.367299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:14.367716 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:14.367908 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:14.368185 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:29:14.368309 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260925 15:29:14.370208 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:14.371349 21945 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:14.372228 21945 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:14.380285 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:14.371906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54233 (local address 127.0.0.1:36875)
0925 15:29:14.371948 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:14.371950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:14.372077 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:29:14.372177 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:14.372178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:14.372248 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:14.372309 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:14.373060 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:14.373922 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:14.374988 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:14.375332 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:14.375400 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:14.377107 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:14.377108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:14.377109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:14.377117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:14.378942 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:14.379432 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:14.379433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:14.379433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:14.379493 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:14.379732 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:14.379733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:14.379733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:14.379866 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:14.379984 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:14.380118 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 15:29:14.380182 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:14.384727 21945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 656c402bac05413f8be78350d945d35b: Generated new TSK 0
I20260925 15:29:14.385004 21945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:14.387646 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21865
W20260925 15:29:14.392509 21850 connection.cc:535] client connection to 127.0.0.1:36875 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36875): Connection reset by peer (error 104)
2026-09-25T15:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21964](info): set up 2 sockets
Sep 25 15:29:14 dist-test-slave-hvv5 krb5kdc[21964](info): commencing operation
krb5kdc: starting...
W20260925 15:29:15.441311 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:15 dist-test-slave-hvv5 krb5kdc[21964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:15.576778 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:15.660133 21977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:15.660401 21977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:15.660439 21977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:15.661672 21977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:15.661875 21977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:15.661906 21977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:15.661940 21977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:15.661965 21977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:15.663645 21977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:15.664016 21977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:15.664275 21977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:15.669600 21983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:15.669606 21982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:15.669618 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:15.669896 21977 server_base.cc:1034] running on GCE node
I20260925 15:29:15.670187 21977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:15.670531 21977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:15.671742 21977 hybrid_clock.cc:648] HybridClock initialized: now 1790350155671682 us; error 51 us; skew 500 ppm
Sep 25 15:29:15 dist-test-slave-hvv5 krb5kdc[21964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:15.674433 21977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:15.674932 21977 webserver.cc:460] Webserver started at http://127.0.0.1:46825/ using document root <none> and password file <none>
I20260925 15:29:15.675150 21977 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:15.675192 21977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:15.675287 21977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:15.676306 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "791baa91c96f4caa8f2a9e6e01fd5161"
format_stamp: "Formatted at 2026-09-25 15:29:15 on dist-test-slave-hvv5"
server_key: "9ec7e3f9dcb15a4e0bf08c57af9fa57c"
server_key_iv: "46a1af52006c3e30939e0688f0bc9609"
server_key_version: "encryptionkey@0"
I20260925 15:29:15.676755 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "791baa91c96f4caa8f2a9e6e01fd5161"
format_stamp: "Formatted at 2026-09-25 15:29:15 on dist-test-slave-hvv5"
server_key: "9ec7e3f9dcb15a4e0bf08c57af9fa57c"
server_key_iv: "46a1af52006c3e30939e0688f0bc9609"
server_key_version: "encryptionkey@0"
I20260925 15:29:15.678063 21977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:15.678776 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:15.678949 21977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:29:15.679059 21977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "791baa91c96f4caa8f2a9e6e01fd5161"
format_stamp: "Formatted at 2026-09-25 15:29:15 on dist-test-slave-hvv5"
server_key: "9ec7e3f9dcb15a4e0bf08c57af9fa57c"
server_key_iv: "46a1af52006c3e30939e0688f0bc9609"
server_key_version: "encryptionkey@0"
I20260925 15:29:15.679136 21977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:15.705403 21977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:15.707942 21977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:15.708118 21977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:15.712442 21977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41017
I20260925 15:29:15.712502 22044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41017 every 8 connection(s)
I20260925 15:29:15.712914 21977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:15.713972 22045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:15.716392 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161: Bootstrap starting.
I20260925 15:29:15.717061 22045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:15.717356 22045 log.cc:824] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:15.718151 22045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161: No bootstrap required, opened a new log
I20260925 15:29:15.719798 22045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260925 15:29:15.720224 22045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:15.720266 22045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791baa91c96f4caa8f2a9e6e01fd5161, State: Initialized, Role: FOLLOWER
I20260925 15:29:15.720495 22045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260925 15:29:15.720638 22045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:15.720743 22045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:15.720806 22045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:15.721473 22045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260925 15:29:15.721597 22045 leader_election.cc:302] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791baa91c96f4caa8f2a9e6e01fd5161; no voters:
I20260925 15:29:15.721801 22045 leader_election.cc:288] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:15.721839 22048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:15.722023 22048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [term 1 LEADER]: Becoming Leader. State: Replica: 791baa91c96f4caa8f2a9e6e01fd5161, State: Running, Role: LEADER
I20260925 15:29:15.722098 22045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:15.722167 22048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } }
I20260925 15:29:15.722558 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 21977
I20260925 15:29:15.722671 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:15.722904 1677 external_mini_cluster.cc:1428] Setting key b4edc9d3f69b706421daa67d85b58f56
I20260925 15:29:15.723382 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791baa91c96f4caa8f2a9e6e01fd5161. Latest consensus state: current_term: 1 leader_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } } }
I20260925 15:29:15.723409 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791baa91c96f4caa8f2a9e6e01fd5161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41017 } } }
I20260925 15:29:15.723487 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:15.723492 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:15.723701 22064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:15.724431 22064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:15.726234 22064 catalog_manager.cc:1347] Generated new cluster ID: 27862913d84446678fc563d15295f278
I20260925 15:29:15.726287 22064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:15.729682 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:15.723606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46039 (local address 127.0.0.1:41017)
0925 15:29:15.723841 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:15.723851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:15.724049 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:29:15.724241 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:15.724250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:15.724318 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:15.724474 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:15.724725 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:15.725945 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:15.727275 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:15.727936 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:15.728071 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:15.728961 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:15.728965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:15.728966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:15.729185 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:15.729225 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:15.729240 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:15.729324 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:15.729426 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:29:15.729455 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260925 15:29:15.730872 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:15.736575 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:15.732571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50585 (local address 127.0.0.1:41017)
0925 15:29:15.732609 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:15.732611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:15.732770 (+ 159us) server_negotiation.cc:407] Connection header received
0925 15:29:15.732869 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:15.732870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:15.732881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:15.732912 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:15.733132 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:15.733915 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:15.735009 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:15.735324 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:15.735379 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:15.736240 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:15.736241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:15.736242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:15.736247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:15.736263 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:15.736267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:15.736300 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:15.736448 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:29:15.736463 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:29:15.746317 22064 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:15.746974 22064 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:15.759230 22064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791baa91c96f4caa8f2a9e6e01fd5161: Generated new TSK 0
I20260925 15:29:15.759481 22064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:15.762729 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 21977
W20260925 15:29:15.767632 21962 connection.cc:535] client connection to 127.0.0.1:41017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41017): Connection reset by peer (error 104)
2026-09-25T15:29:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:15 dist-test-slave-hvv5 krb5kdc[22076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:15 dist-test-slave-hvv5 krb5kdc[22076](info): set up 2 sockets
Sep 25 15:29:15 dist-test-slave-hvv5 krb5kdc[22076](info): commencing operation
krb5kdc: starting...
W20260925 15:29:16.840672 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:16 dist-test-slave-hvv5 krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:16.964306 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:17.047326 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:17.047588 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:17.047631 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:17.048863 22089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:17.048938 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:17.048962 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:17.048988 22089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:17.049021 22089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:17.050554 22089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:17.050832 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:17.051070 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:17.056214 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:17.056222 22095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:17.056255 22089 server_base.cc:1034] running on GCE node
W20260925 15:29:17.056233 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:17.056699 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:17.057013 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:17.058188 22089 hybrid_clock.cc:648] HybridClock initialized: now 1790350157058184 us; error 38 us; skew 500 ppm
Sep 25 15:29:17 dist-test-slave-hvv5 krb5kdc[22076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:17.060863 22089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:17.061278 22089 webserver.cc:460] Webserver started at http://127.0.0.1:37301/ using document root <none> and password file <none>
I20260925 15:29:17.061478 22089 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:17.061520 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:17.061604 22089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:17.062644 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "96ca3da310db4a9a91b545c4ea729fc5"
format_stamp: "Formatted at 2026-09-25 15:29:17 on dist-test-slave-hvv5"
server_key: "08688b72b2bcd299dd5881f618c3ba27"
server_key_iv: "4c06fda052b102202cb95b63c15cb581"
server_key_version: "encryptionkey@0"
I20260925 15:29:17.062939 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "96ca3da310db4a9a91b545c4ea729fc5"
format_stamp: "Formatted at 2026-09-25 15:29:17 on dist-test-slave-hvv5"
server_key: "08688b72b2bcd299dd5881f618c3ba27"
server_key_iv: "4c06fda052b102202cb95b63c15cb581"
server_key_version: "encryptionkey@0"
I20260925 15:29:17.064227 22089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 15:29:17.065011 22104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:17.065187 22089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:17.065279 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "96ca3da310db4a9a91b545c4ea729fc5"
format_stamp: "Formatted at 2026-09-25 15:29:17 on dist-test-slave-hvv5"
server_key: "08688b72b2bcd299dd5881f618c3ba27"
server_key_iv: "4c06fda052b102202cb95b63c15cb581"
server_key_version: "encryptionkey@0"
I20260925 15:29:17.065357 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:17.069599 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:17.071997 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:17.072160 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:17.076190 22089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36249
I20260925 15:29:17.076221 22156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36249 every 8 connection(s)
I20260925 15:29:17.076669 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:17.077693 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:17.079242 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22089
I20260925 15:29:17.079378 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:17.079599 1677 external_mini_cluster.cc:1428] Setting key 2242a1589896f8b3f772abdc32e9900d
I20260925 15:29:17.080246 22157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5: Bootstrap starting.
I20260925 15:29:17.080940 22157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:17.081230 22157 log.cc:824] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:17.081928 22157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5: No bootstrap required, opened a new log
I20260925 15:29:17.082360 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:17.080193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54583 (local address 127.0.0.1:36249)
0925 15:29:17.080393 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:17.080403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:17.080437 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:29:17.080998 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:17.081005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:17.081062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:17.081248 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:17.081945 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:17.081950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:17.081950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:17.081954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:17.081994 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:17.082030 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:17.082111 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:17.082151 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 15:29:17.082193 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260925 15:29:17.083386 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:17.084379 22157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260925 15:29:17.084606 22157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:17.084661 22157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ca3da310db4a9a91b545c4ea729fc5, State: Initialized, Role: FOLLOWER
I20260925 15:29:17.084787 22157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260925 15:29:17.084900 22157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:17.084976 22157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:17.085060 22157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:17.085631 22157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260925 15:29:17.085752 22157 leader_election.cc:302] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96ca3da310db4a9a91b545c4ea729fc5; no voters:
I20260925 15:29:17.085916 22157 leader_election.cc:288] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:17.085969 22167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:17.086077 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:17.084844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52449 (local address 127.0.0.1:36249)
0925 15:29:17.084880 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:17.084881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:17.085005 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:29:17.085081 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:17.085081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:17.085087 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:17.085136 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:17.085798 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:17.085800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:17.085800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:17.085805 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:17.085820 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:17.085825 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:17.085877 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:17.085973 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 15:29:17.085988 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:29:17.086253 22157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:17.086473 22167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 96ca3da310db4a9a91b545c4ea729fc5, State: Running, Role: LEADER
I20260925 15:29:17.086624 22167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260925 15:29:17.087038 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96ca3da310db4a9a91b545c4ea729fc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260925 15:29:17.087030 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96ca3da310db4a9a91b545c4ea729fc5. Latest consensus state: current_term: 1 leader_uuid: "96ca3da310db4a9a91b545c4ea729fc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96ca3da310db4a9a91b545c4ea729fc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260925 15:29:17.087132 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:17.087138 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:17.087360 22176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:17.087870 22176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:17.089514 22176 catalog_manager.cc:1347] Generated new cluster ID: 27bb25d48eef4a778b9d4d82ff801eb4
I20260925 15:29:17.089572 22176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:17.112512 22176 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:17.112833 22176 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:17.119206 22176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96ca3da310db4a9a91b545c4ea729fc5: Generated new TSK 0
I20260925 15:29:17.119433 22176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:17.124105 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22089
2026-09-25T15:29:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:17 dist-test-slave-hvv5 krb5kdc[22188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:17 dist-test-slave-hvv5 krb5kdc[22188](info): set up 2 sockets
Sep 25 15:29:17 dist-test-slave-hvv5 krb5kdc[22188](info): commencing operation
krb5kdc: starting...
W20260925 15:29:18.172951 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:18.307816 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:18.391489 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:18.391711 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:18.391737 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:18.393040 22201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:18.393087 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:18.393102 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:18.393155 22201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:18.393181 22201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:18.394706 22201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:18.394981 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:18.395226 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:18.400791 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:18.400791 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:18.400933 22201 server_base.cc:1034] running on GCE node
W20260925 15:29:18.400822 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:18.401391 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:18.401678 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:18.402963 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790350158402941 us; error 38 us; skew 500 ppm
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:18.405840 22201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:18.406450 22201 webserver.cc:460] Webserver started at http://127.0.0.1:42787/ using document root <none> and password file <none>
I20260925 15:29:18.406709 22201 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:18.406787 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:18.406950 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:18.408131 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "1ed583f74d604d9b9ca57e777f2d5e45"
format_stamp: "Formatted at 2026-09-25 15:29:18 on dist-test-slave-hvv5"
server_key: "34f32ab26d1af855f8c41c5acf96afcd"
server_key_iv: "5f1e3ecfa43db076f364410d91b59db1"
server_key_version: "encryptionkey@0"
I20260925 15:29:18.408653 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "1ed583f74d604d9b9ca57e777f2d5e45"
format_stamp: "Formatted at 2026-09-25 15:29:18 on dist-test-slave-hvv5"
server_key: "34f32ab26d1af855f8c41c5acf96afcd"
server_key_iv: "5f1e3ecfa43db076f364410d91b59db1"
server_key_version: "encryptionkey@0"
I20260925 15:29:18.410260 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:18.411180 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:18.411396 22201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:18.411507 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "1ed583f74d604d9b9ca57e777f2d5e45"
format_stamp: "Formatted at 2026-09-25 15:29:18 on dist-test-slave-hvv5"
server_key: "34f32ab26d1af855f8c41c5acf96afcd"
server_key_iv: "5f1e3ecfa43db076f364410d91b59db1"
server_key_version: "encryptionkey@0"
I20260925 15:29:18.411598 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:18.429845 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:18.432386 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:18.432587 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:18.437107 22201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44781
I20260925 15:29:18.437166 22268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44781 every 8 connection(s)
I20260925 15:29:18.437582 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:18.438658 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:18.441250 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45: Bootstrap starting.
I20260925 15:29:18.441891 22269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:18.442201 22269 log.cc:824] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:18.443084 22269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45: No bootstrap required, opened a new log
I20260925 15:29:18.443257 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22201
I20260925 15:29:18.443368 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:18.443557 1677 external_mini_cluster.cc:1428] Setting key 1ed900984730d27fd2ee3670e5bc85e7
I20260925 15:29:18.445322 22269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260925 15:29:18.445472 22269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:18.445495 22269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed583f74d604d9b9ca57e777f2d5e45, State: Initialized, Role: FOLLOWER
I20260925 15:29:18.445621 22269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260925 15:29:18.445704 22269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:18.445740 22269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:18.445847 22269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:18.447129 22269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260925 15:29:18.447240 22269 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed583f74d604d9b9ca57e777f2d5e45; no voters:
I20260925 15:29:18.447417 22269 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:18.447494 22274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:18.447618 22274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed583f74d604d9b9ca57e777f2d5e45, State: Running, Role: LEADER
I20260925 15:29:18.447707 22269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:18.447793 22274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260925 15:29:18.448590 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed583f74d604d9b9ca57e777f2d5e45. Latest consensus state: current_term: 1 leader_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260925 15:29:18.449385 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:18.449677 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed583f74d604d9b9ca57e777f2d5e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260925 15:29:18.449756 22286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:18.450518 22286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:18.450637 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45 [sys.catalog]: This master's current role is: LEADER
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:29:18.452675 22286 catalog_manager.cc:1347] Generated new cluster ID: edf450af6d2241208f2d51f9251e2568
I20260925 15:29:18.452746 22286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:18.456753 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:18.444207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48477 (local address 127.0.0.1:44781)
0925 15:29:18.444422 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:18.444430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:18.444458 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:29:18.445327 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:18.445336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:18.445396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:18.445680 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:18.446631 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:18.447978 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:18.449047 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:18.449624 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:18.450165 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:18.452806 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:18.452811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:18.452811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:18.452823 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:18.454947 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:18.455376 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:18.455379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:18.455380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:18.455518 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:18.455729 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:18.455730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:18.455730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:18.456059 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:18.456253 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:18.456326 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:29:18.456483 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260925 15:29:18.457805 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:18.460881 22286 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:18.461601 22286 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:18.467100 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:18.459416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33665 (local address 127.0.0.1:44781)
0925 15:29:18.459453 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:18.459455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:18.459577 (+ 122us) server_negotiation.cc:407] Connection header received
0925 15:29:18.459629 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:18.459630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:18.459640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:18.459709 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:18.460357 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:18.461143 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:18.462130 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:18.462432 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:18.462506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:18.463911 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:18.463913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:18.463913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:18.463920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:18.465625 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:18.466044 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:18.466045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:18.466046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:18.466109 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:18.466394 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:18.466396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:18.466396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:18.466563 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:18.466677 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:18.466871 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:29:18.466998 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:18.469283 22286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed583f74d604d9b9ca57e777f2d5e45: Generated new TSK 0
I20260925 15:29:18.469506 22286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:18.471973 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22201
2026-09-25T15:29:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22300](info): set up 2 sockets
Sep 25 15:29:18 dist-test-slave-hvv5 krb5kdc[22300](info): commencing operation
krb5kdc: starting...
W20260925 15:29:19.528798 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:19 dist-test-slave-hvv5 krb5kdc[22300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:19.656760 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:19.742194 22313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:19.742403 22313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:19.742426 22313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:19.743558 22313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:19.743598 22313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:19.743613 22313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:19.743626 22313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:19.743638 22313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:19.745219 22313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:19.745434 22313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:19.745621 22313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:19.750885 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:19.750784 22318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:19.750771 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:19.750984 22313 server_base.cc:1034] running on GCE node
I20260925 15:29:19.751397 22313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:19.751704 22313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:19.752888 22313 hybrid_clock.cc:648] HybridClock initialized: now 1790350159752877 us; error 48 us; skew 500 ppm
Sep 25 15:29:19 dist-test-slave-hvv5 krb5kdc[22300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:19.761562 22313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:19.762197 22313 webserver.cc:460] Webserver started at http://127.0.0.1:34563/ using document root <none> and password file <none>
I20260925 15:29:19.762486 22313 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:19.762566 22313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:19.762799 22313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:19.764011 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "a091592ff1fc4ec7930dbaee20457199"
format_stamp: "Formatted at 2026-09-25 15:29:19 on dist-test-slave-hvv5"
server_key: "59ef3f9ecf84d5b7410d2bff48df0f46"
server_key_iv: "c639090ebd1e273fc7dfd0ae927976e1"
server_key_version: "encryptionkey@0"
I20260925 15:29:19.764416 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "a091592ff1fc4ec7930dbaee20457199"
format_stamp: "Formatted at 2026-09-25 15:29:19 on dist-test-slave-hvv5"
server_key: "59ef3f9ecf84d5b7410d2bff48df0f46"
server_key_iv: "c639090ebd1e273fc7dfd0ae927976e1"
server_key_version: "encryptionkey@0"
I20260925 15:29:19.766134 22313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:19.766969 22328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:19.767151 22313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:19.767266 22313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "a091592ff1fc4ec7930dbaee20457199"
format_stamp: "Formatted at 2026-09-25 15:29:19 on dist-test-slave-hvv5"
server_key: "59ef3f9ecf84d5b7410d2bff48df0f46"
server_key_iv: "c639090ebd1e273fc7dfd0ae927976e1"
server_key_version: "encryptionkey@0"
I20260925 15:29:19.767347 22313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:19.779163 22313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:19.781313 22313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:19.781484 22313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:19.785583 22313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45993
I20260925 15:29:19.785619 22380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45993 every 8 connection(s)
I20260925 15:29:19.785998 22313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:19.787040 22381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:19.789362 22381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: Bootstrap starting.
I20260925 15:29:19.789958 22381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:19.790239 22381 log.cc:824] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:19.790958 22381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: No bootstrap required, opened a new log
I20260925 15:29:19.792333 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22313
I20260925 15:29:19.792454 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:19.792701 1677 external_mini_cluster.cc:1428] Setting key 73c515b4e5aeff9d6b2701d562f5256c
I20260925 15:29:19.792917 22381 raft_consensus.cc:348] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } }
I20260925 15:29:19.793071 22381 raft_consensus.cc:374] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:19.793133 22381 raft_consensus.cc:729] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a091592ff1fc4ec7930dbaee20457199, State: Initialized, Role: FOLLOWER
I20260925 15:29:19.793287 22381 consensus_queue.cc:260] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } }
I20260925 15:29:19.793403 22381 raft_consensus.cc:388] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:19.793452 22381 raft_consensus.cc:482] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:19.793533 22381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:19.794428 22381 raft_consensus.cc:504] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } }
I20260925 15:29:19.794516 22381 leader_election.cc:302] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a091592ff1fc4ec7930dbaee20457199; no voters:
I20260925 15:29:19.794697 22381 leader_election.cc:288] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:19.794727 22386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:19.794835 22386 raft_consensus.cc:686] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [term 1 LEADER]: Becoming Leader. State: Replica: a091592ff1fc4ec7930dbaee20457199, State: Running, Role: LEADER
I20260925 15:29:19.794965 22381 sys_catalog.cc:564] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:19.794966 22386 consensus_queue.cc:237] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } }
I20260925 15:29:19.796195 22388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a091592ff1fc4ec7930dbaee20457199. Latest consensus state: current_term: 1 leader_uuid: "a091592ff1fc4ec7930dbaee20457199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } } }
I20260925 15:29:19.796288 22388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:19.796401 22387 sys_catalog.cc:455] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a091592ff1fc4ec7930dbaee20457199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a091592ff1fc4ec7930dbaee20457199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45993 } } }
I20260925 15:29:19.796612 22387 sys_catalog.cc:458] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199 [sys.catalog]: This master's current role is: LEADER
W20260925 15:29:19.796898 22401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:19.796990 22401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:19.797112 22402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:19.797312 22402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:19.798910 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:19.793234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45107 (local address 127.0.0.1:45993)
0925 15:29:19.793438 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:19.793448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:19.793481 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:29:19.793620 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:19.793627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:19.793680 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:19.793811 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:19.794080 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:19.795358 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:19.796576 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:19.797065 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:19.797204 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:19.798122 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:19.798126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:19.798126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:19.798380 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:19.798419 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:19.798435 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:19.798510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:19.798605 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:29:19.798692 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260925 15:29:19.799321 22402 catalog_manager.cc:1347] Generated new cluster ID: 5e04be4d5eb84fa1b166e877c8992050
I20260925 15:29:19.799376 22402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:19.799713 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:19.804632 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:19.801120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35025 (local address 127.0.0.1:45993)
0925 15:29:19.801153 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:19.801155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:19.801243 (+ 88us) server_negotiation.cc:407] Connection header received
0925 15:29:19.801297 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:19.801298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:19.801319 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:19.801384 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:19.801509 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:19.802312 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:19.803316 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:19.803611 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:19.803686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:19.804349 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:19.804351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:19.804351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:19.804356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:19.804371 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:19.804375 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:19.804438 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:19.804503 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:29:19.804527 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":17}
I20260925 15:29:19.820339 22402 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:19.821022 22402 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:19.828423 22402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a091592ff1fc4ec7930dbaee20457199: Generated new TSK 0
I20260925 15:29:19.828658 22402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:19.834731 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22313
2026-09-25T15:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:19 dist-test-slave-hvv5 krb5kdc[22412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:19 dist-test-slave-hvv5 krb5kdc[22412](info): set up 2 sockets
Sep 25 15:29:19 dist-test-slave-hvv5 krb5kdc[22412](info): commencing operation
krb5kdc: starting...
W20260925 15:29:20.896941 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:20 dist-test-slave-hvv5 krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:21.025147 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:21.107751 22425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:21.108009 22425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:21.108052 22425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:21.109340 22425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:21.109417 22425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:21.109441 22425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:21.109463 22425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:21.109493 22425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:21.111035 22425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:21.111334 22425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:21.111588 22425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:21.116842 22430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:21.116842 22433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:21.116955 22425 server_base.cc:1034] running on GCE node
W20260925 15:29:21.116856 22431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:21.117533 22425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:21.117925 22425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:21.119114 22425 hybrid_clock.cc:648] HybridClock initialized: now 1790350161119118 us; error 47 us; skew 500 ppm
Sep 25 15:29:21 dist-test-slave-hvv5 krb5kdc[22412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:21.121846 22425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:21.122304 22425 webserver.cc:460] Webserver started at http://127.0.0.1:40587/ using document root <none> and password file <none>
I20260925 15:29:21.122568 22425 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:21.122640 22425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:21.122793 22425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:21.123940 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "4877c010f5634748b1cb2944d6405405"
format_stamp: "Formatted at 2026-09-25 15:29:21 on dist-test-slave-hvv5"
server_key: "62e0d1eb977b840e0062038e653fe21e"
server_key_iv: "fc64914859bc48affcfb78d20caff458"
server_key_version: "encryptionkey@0"
I20260925 15:29:21.124307 22425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "4877c010f5634748b1cb2944d6405405"
format_stamp: "Formatted at 2026-09-25 15:29:21 on dist-test-slave-hvv5"
server_key: "62e0d1eb977b840e0062038e653fe21e"
server_key_iv: "fc64914859bc48affcfb78d20caff458"
server_key_version: "encryptionkey@0"
I20260925 15:29:21.125751 22425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:21.126600 22440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:21.126812 22425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:21.126935 22425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "4877c010f5634748b1cb2944d6405405"
format_stamp: "Formatted at 2026-09-25 15:29:21 on dist-test-slave-hvv5"
server_key: "62e0d1eb977b840e0062038e653fe21e"
server_key_iv: "fc64914859bc48affcfb78d20caff458"
server_key_version: "encryptionkey@0"
I20260925 15:29:21.127019 22425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:21.130545 22425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:21.132725 22425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:21.132889 22425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:21.136955 22425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36085
I20260925 15:29:21.137017 22492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36085 every 8 connection(s)
I20260925 15:29:21.137382 22425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:21.138427 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:21.140169 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22425
I20260925 15:29:21.140298 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:21.140483 1677 external_mini_cluster.cc:1428] Setting key 48cafbc1bd51ae242a4829a44f15c834
I20260925 15:29:21.141135 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: Bootstrap starting.
I20260925 15:29:21.141767 22493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:21.142069 22493 log.cc:824] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:21.142952 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: No bootstrap required, opened a new log
I20260925 15:29:21.143051 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:21.141053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40729 (local address 127.0.0.1:36085)
0925 15:29:21.141239 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:21.141247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:21.141274 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:29:21.141695 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:21.141702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:21.141728 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:21.141930 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:21.142612 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:21.142616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:21.142616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:21.142621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:21.142668 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:21.142684 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:21.142784 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:21.142814 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 15:29:21.142856 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260925 15:29:21.144022 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:21.145342 22493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } }
I20260925 15:29:21.145566 22493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:21.145622 22493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4877c010f5634748b1cb2944d6405405, State: Initialized, Role: FOLLOWER
I20260925 15:29:21.145756 22493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } }
I20260925 15:29:21.145879 22493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:21.145918 22493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:21.145982 22493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:21.146636 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:21.145490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53553 (local address 127.0.0.1:36085)
0925 15:29:21.145534 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:21.145536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:21.145588 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:29:21.145651 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:21.145651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:21.145656 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:21.145713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:21.146454 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:21.146455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:21.146455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:21.146459 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:21.146473 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:21.146477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:21.146519 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:21.146581 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:29:21.146593 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:29:21.146792 22493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } }
I20260925 15:29:21.146916 22493 leader_election.cc:302] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4877c010f5634748b1cb2944d6405405; no voters:
I20260925 15:29:21.147084 22493 leader_election.cc:288] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:21.147107 22503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:21.147217 22503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [term 1 LEADER]: Becoming Leader. State: Replica: 4877c010f5634748b1cb2944d6405405, State: Running, Role: LEADER
I20260925 15:29:21.147344 22493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:21.147333 22503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } }
I20260925 15:29:21.147734 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4877c010f5634748b1cb2944d6405405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } } }
I20260925 15:29:21.147746 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4877c010f5634748b1cb2944d6405405. Latest consensus state: current_term: 1 leader_uuid: "4877c010f5634748b1cb2944d6405405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4877c010f5634748b1cb2944d6405405" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36085 } } }
I20260925 15:29:21.147903 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:21.147918 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405 [sys.catalog]: This master's current role is: LEADER
W20260925 15:29:21.149099 22519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:21.149145 22519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:21.149183 22512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:21.149392 22512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:21.150823 22512 catalog_manager.cc:1347] Generated new cluster ID: 94bfe1d05e8f4e65afc32a44a5ea76b7
I20260925 15:29:21.150872 22512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:21.162880 22512 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:21.163075 22512 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:21.167094 22512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4877c010f5634748b1cb2944d6405405: Generated new TSK 0
I20260925 15:29:21.167297 22512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:21.170471 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22425
2026-09-25T15:29:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:21 dist-test-slave-hvv5 krb5kdc[22524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:21 dist-test-slave-hvv5 krb5kdc[22524](info): set up 2 sockets
Sep 25 15:29:21 dist-test-slave-hvv5 krb5kdc[22524](info): commencing operation
krb5kdc: starting...
W20260925 15:29:22.249706 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:22.384021 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:22.469324 22537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:22.469554 22537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:22.469580 22537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:22.470870 22537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:22.470922 22537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:22.470937 22537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:22.470950 22537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:22.470963 22537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:22.472630 22537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:22.472919 22537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:22.473181 22537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:22.478806 22543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:22.478808 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:22.479035 22537 server_base.cc:1034] running on GCE node
W20260925 15:29:22.478785 22542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:22.479447 22537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:22.479730 22537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:22.480872 22537 hybrid_clock.cc:648] HybridClock initialized: now 1790350162480855 us; error 32 us; skew 500 ppm
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:22.483651 22537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:22.484091 22537 webserver.cc:460] Webserver started at http://127.0.0.1:36855/ using document root <none> and password file <none>
I20260925 15:29:22.484314 22537 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:22.484359 22537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:22.484450 22537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:22.485580 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "a92fb5a990254afdae3efc2af0ec6387"
format_stamp: "Formatted at 2026-09-25 15:29:22 on dist-test-slave-hvv5"
server_key: "33b20ca23288e5efa731ff6dbb3a6e8d"
server_key_iv: "9498bdb5ff2d0ad11890bc66bf11df30"
server_key_version: "encryptionkey@0"
I20260925 15:29:22.485889 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "a92fb5a990254afdae3efc2af0ec6387"
format_stamp: "Formatted at 2026-09-25 15:29:22 on dist-test-slave-hvv5"
server_key: "33b20ca23288e5efa731ff6dbb3a6e8d"
server_key_iv: "9498bdb5ff2d0ad11890bc66bf11df30"
server_key_version: "encryptionkey@0"
I20260925 15:29:22.487303 22537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:22.488099 22552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:22.488358 22537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:22.488480 22537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "a92fb5a990254afdae3efc2af0ec6387"
format_stamp: "Formatted at 2026-09-25 15:29:22 on dist-test-slave-hvv5"
server_key: "33b20ca23288e5efa731ff6dbb3a6e8d"
server_key_iv: "9498bdb5ff2d0ad11890bc66bf11df30"
server_key_version: "encryptionkey@0"
I20260925 15:29:22.488592 22537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:22.508036 22537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:22.510313 22537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:22.510509 22537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:22.514891 22537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43943
I20260925 15:29:22.514925 22604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43943 every 8 connection(s)
I20260925 15:29:22.515307 22537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:22.516423 22605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:22.518867 22605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: Bootstrap starting.
I20260925 15:29:22.519474 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22537
I20260925 15:29:22.519778 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:22.519824 22605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:22.520010 1677 external_mini_cluster.cc:1428] Setting key 1998268818a2cfc58d1bd547911044a7
I20260925 15:29:22.520206 22605 log.cc:824] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:22.521198 22605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: No bootstrap required, opened a new log
I20260925 15:29:22.523674 22605 raft_consensus.cc:348] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 15:29:22.523877 22605 raft_consensus.cc:374] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:22.523916 22605 raft_consensus.cc:729] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92fb5a990254afdae3efc2af0ec6387, State: Initialized, Role: FOLLOWER
I20260925 15:29:22.524055 22605 consensus_queue.cc:260] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 15:29:22.524312 22605 raft_consensus.cc:388] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:22.524428 22605 raft_consensus.cc:482] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:22.524513 22605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:22.525329 22605 raft_consensus.cc:504] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 15:29:22.525458 22605 leader_election.cc:302] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a92fb5a990254afdae3efc2af0ec6387; no voters:
I20260925 15:29:22.525676 22605 leader_election.cc:288] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:22.525734 22610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:22.525974 22605 sys_catalog.cc:564] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:22.525974 22610 raft_consensus.cc:686] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [term 1 LEADER]: Becoming Leader. State: Replica: a92fb5a990254afdae3efc2af0ec6387, State: Running, Role: LEADER
I20260925 15:29:22.526124 22610 consensus_queue.cc:237] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260925 15:29:22.526510 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a92fb5a990254afdae3efc2af0ec6387. Latest consensus state: current_term: 1 leader_uuid: "a92fb5a990254afdae3efc2af0ec6387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260925 15:29:22.526667 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:22.526778 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a92fb5a990254afdae3efc2af0ec6387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92fb5a990254afdae3efc2af0ec6387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260925 15:29:22.527854 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387 [sys.catalog]: This master's current role is: LEADER
W20260925 15:29:22.528587 22625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:22.528648 22625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:22.528703 22626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:22.528884 22626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:29:22.531140 22626 catalog_manager.cc:1347] Generated new cluster ID: 395dfcc185c4495e8ab82cc9ed4bf46a
I20260925 15:29:22.531199 22626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:22.534267 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:22.520651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:60347 (local address 127.0.0.1:43943)
0925 15:29:22.520831 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:22.520842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:22.520878 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:29:22.521538 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:22.521548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:22.521620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:22.521798 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:22.522515 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:22.524027 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:22.525191 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:22.525854 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:22.525992 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:22.529620 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:22.529625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:22.529626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:22.529641 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:22.531970 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:22.532506 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:22.532510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:22.532510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:22.532690 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:22.533083 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:22.533084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:22.533085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:22.533405 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:22.533794 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:22.533902 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:29:22.534030 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260925 15:29:22.535667 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:22.544837 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:22.537230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:55351 (local address 127.0.0.1:43943)
0925 15:29:22.537329 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:22.537331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:22.537383 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:29:22.537453 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:22.537454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:22.537464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:22.537520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:22.538091 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:22.538609 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:22.539351 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:22.539586 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:22.539678 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:22.541108 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:22.541109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:22.541109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:22.541118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:22.543020 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:22.543447 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:22.543448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:22.543449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:22.543514 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:22.544094 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:22.544095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:22.544096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:22.544227 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:22.544352 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:22.544522 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:29:22.544634 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:29:22.548834 22626 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:22.549436 22626 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:22.553452 22626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a92fb5a990254afdae3efc2af0ec6387: Generated new TSK 0
I20260925 15:29:22.553678 22626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:22.560065 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22537
W20260925 15:29:22.564015 22522 connection.cc:535] client connection to 127.0.0.1:43943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43943): Connection reset by peer (error 104)
2026-09-25T15:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22636](info): set up 2 sockets
Sep 25 15:29:22 dist-test-slave-hvv5 krb5kdc[22636](info): commencing operation
krb5kdc: starting...
W20260925 15:29:23.637259 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:23 dist-test-slave-hvv5 krb5kdc[22636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:23.821653 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:23.912106 22649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:23.912667 22649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:23.912760 22649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:23.914247 22649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:23.914446 22649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:23.914518 22649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:23.914558 22649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:23.914590 22649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:23.916872 22649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:23.917340 22649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:23.917766 22649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:23.923749 22654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:23.923914 22655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:23.923868 22649 server_base.cc:1034] running on GCE node
W20260925 15:29:23.923837 22657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:23.924458 22649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:23.925226 22649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:23.926456 22649 hybrid_clock.cc:648] HybridClock initialized: now 1790350163926431 us; error 42 us; skew 500 ppm
Sep 25 15:29:23 dist-test-slave-hvv5 krb5kdc[22636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:23.929605 22649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:23.930162 22649 webserver.cc:460] Webserver started at http://127.0.0.1:39713/ using document root <none> and password file <none>
I20260925 15:29:23.930474 22649 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:23.930543 22649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:23.930696 22649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:23.931828 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "ddf562aad009468985bf36f2888bd021"
format_stamp: "Formatted at 2026-09-25 15:29:23 on dist-test-slave-hvv5"
server_key: "0f5ae0b9a32a3b8b6a54bdd7e8c55ad2"
server_key_iv: "5fddb29af122432ec3b0c232533c3361"
server_key_version: "encryptionkey@0"
I20260925 15:29:23.932224 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "ddf562aad009468985bf36f2888bd021"
format_stamp: "Formatted at 2026-09-25 15:29:23 on dist-test-slave-hvv5"
server_key: "0f5ae0b9a32a3b8b6a54bdd7e8c55ad2"
server_key_iv: "5fddb29af122432ec3b0c232533c3361"
server_key_version: "encryptionkey@0"
I20260925 15:29:23.933817 22649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:23.934866 22664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:23.935190 22649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:23.935356 22649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "ddf562aad009468985bf36f2888bd021"
format_stamp: "Formatted at 2026-09-25 15:29:23 on dist-test-slave-hvv5"
server_key: "0f5ae0b9a32a3b8b6a54bdd7e8c55ad2"
server_key_iv: "5fddb29af122432ec3b0c232533c3361"
server_key_version: "encryptionkey@0"
I20260925 15:29:23.935456 22649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:23.947929 22649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:23.950520 22649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:23.950805 22649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:23.955420 22649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34845
I20260925 15:29:23.955507 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34845 every 8 connection(s)
I20260925 15:29:23.955925 22649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:23.957105 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:23.958041 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22649
I20260925 15:29:23.958184 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:23.958395 1677 external_mini_cluster.cc:1428] Setting key 2570ca93890011a1407e97fdc2ef70f8
I20260925 15:29:23.960626 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021: Bootstrap starting.
I20260925 15:29:23.961664 22717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:23.962036 22717 log.cc:824] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:23.962914 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021: No bootstrap required, opened a new log
I20260925 15:29:23.964129 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:23.958993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:56939 (local address 127.0.0.1:34845)
0925 15:29:23.959211 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:23.959220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:23.959250 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:29:23.959726 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:23.959734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:23.959789 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:23.959946 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:23.960180 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:23.961256 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:23.962098 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:23.962554 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:23.962675 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:23.963491 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:23.963495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:23.963496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:23.963672 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:23.963703 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:23.963714 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:23.963811 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:23.963913 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:29:23.963944 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260925 15:29:23.965436 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:23.965498 22717 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260925 15:29:23.965706 22717 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:23.965764 22717 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf562aad009468985bf36f2888bd021, State: Initialized, Role: FOLLOWER
I20260925 15:29:23.965890 22717 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260925 15:29:23.965965 22717 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:23.965986 22717 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:23.966053 22717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:23.966813 22717 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260925 15:29:23.966950 22717 leader_election.cc:302] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf562aad009468985bf36f2888bd021; no voters:
I20260925 15:29:23.967130 22717 leader_election.cc:288] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:23.967329 22728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:23.967434 22717 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:23.967444 22728 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [term 1 LEADER]: Becoming Leader. State: Replica: ddf562aad009468985bf36f2888bd021, State: Running, Role: LEADER
I20260925 15:29:23.967554 22728 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260925 15:29:23.968088 22727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf562aad009468985bf36f2888bd021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260925 15:29:23.968215 22727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:23.968083 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf562aad009468985bf36f2888bd021. Latest consensus state: current_term: 1 leader_uuid: "ddf562aad009468985bf36f2888bd021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf562aad009468985bf36f2888bd021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260925 15:29:23.968410 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:23.968570 22731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:23.969126 22731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:23.971074 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:23.967091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:51349 (local address 127.0.0.1:34845)
0925 15:29:23.967202 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:23.967205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:23.967217 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:29:23.967267 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:23.967268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:23.967277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:23.967347 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:23.967559 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:23.968589 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:23.969705 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:23.969964 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:23.970036 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:23.970783 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:23.970784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:23.970785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:23.970791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:23.970809 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:23.970814 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:23.970890 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:23.970967 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:29:23.970978 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:29:23.971326 22731 catalog_manager.cc:1347] Generated new cluster ID: f4abcf8d91e64304864b0b9e06e89c32
I20260925 15:29:23.971431 22731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:23.980558 22731 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:23.981336 22731 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:23.999789 22731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf562aad009468985bf36f2888bd021: Generated new TSK 0
I20260925 15:29:24.000077 22731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:24.006772 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22649
W20260925 15:29:24.011464 22634 connection.cc:535] client connection to 127.0.0.1:34845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34845): Connection reset by peer (error 104)
2026-09-25T15:29:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:24 dist-test-slave-hvv5 krb5kdc[22748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:24 dist-test-slave-hvv5 krb5kdc[22748](info): set up 2 sockets
Sep 25 15:29:24 dist-test-slave-hvv5 krb5kdc[22748](info): commencing operation
krb5kdc: starting...
W20260925 15:29:25.128896 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:25 dist-test-slave-hvv5 krb5kdc[22748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:25.265479 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:25.355695 22761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:25.355996 22761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:25.356226 22761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:25.357645 22761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:25.357726 22761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:25.357761 22761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:25.357795 22761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:25.357824 22761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:25.360004 22761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:25.360378 22761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:25.360659 22761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:25.366103 22766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:25.366366 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:25.366279 22761 server_base.cc:1034] running on GCE node
W20260925 15:29:25.366081 22767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:25.366808 22761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:25.367177 22761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:25.368342 22761 hybrid_clock.cc:648] HybridClock initialized: now 1790350165368331 us; error 33 us; skew 500 ppm
Sep 25 15:29:25 dist-test-slave-hvv5 krb5kdc[22748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:25.378731 22761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:25.379218 22761 webserver.cc:460] Webserver started at http://127.0.0.1:44465/ using document root <none> and password file <none>
I20260925 15:29:25.379503 22761 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:25.379556 22761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:25.379711 22761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:25.381165 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "df0ec26006c04debb19f030d1a46fdb5"
format_stamp: "Formatted at 2026-09-25 15:29:25 on dist-test-slave-hvv5"
server_key: "a1b9f5a44a13f9a402b7c9e769aa0075"
server_key_iv: "413cba2704e4b8b343ed4fee6c7288db"
server_key_version: "encryptionkey@0"
I20260925 15:29:25.381583 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "df0ec26006c04debb19f030d1a46fdb5"
format_stamp: "Formatted at 2026-09-25 15:29:25 on dist-test-slave-hvv5"
server_key: "a1b9f5a44a13f9a402b7c9e769aa0075"
server_key_iv: "413cba2704e4b8b343ed4fee6c7288db"
server_key_version: "encryptionkey@0"
I20260925 15:29:25.383359 22761 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:29:25.384369 22776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:25.384802 22761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:25.384902 22761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "df0ec26006c04debb19f030d1a46fdb5"
format_stamp: "Formatted at 2026-09-25 15:29:25 on dist-test-slave-hvv5"
server_key: "a1b9f5a44a13f9a402b7c9e769aa0075"
server_key_iv: "413cba2704e4b8b343ed4fee6c7288db"
server_key_version: "encryptionkey@0"
I20260925 15:29:25.384966 22761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:25.388809 22761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:25.391708 22761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:25.391839 22761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:25.396133 22761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41819
I20260925 15:29:25.396339 22828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41819 every 8 connection(s)
I20260925 15:29:25.396648 22761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:25.397781 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:25.400151 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5: Bootstrap starting.
I20260925 15:29:25.400847 22829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:25.401093 22829 log.cc:824] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:25.401736 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22761
I20260925 15:29:25.401870 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:25.401943 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5: No bootstrap required, opened a new log
I20260925 15:29:25.402076 1677 external_mini_cluster.cc:1428] Setting key 8b93df8e6039d38e289de3cd43802a5f
I20260925 15:29:25.404088 22829 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 15:29:25.404266 22829 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:25.404292 22829 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0ec26006c04debb19f030d1a46fdb5, State: Initialized, Role: FOLLOWER
I20260925 15:29:25.404435 22829 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 15:29:25.404685 22829 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:25.404796 22829 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:25.404861 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:25.405951 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:25.402711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:42613 (local address 127.0.0.1:41819)
0925 15:29:25.402922 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:25.402930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:25.402957 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:29:25.404208 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:25.404217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:25.404254 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:25.404499 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:25.405392 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:25.405396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:25.405396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:25.405400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:25.405440 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:25.405451 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:25.405518 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:25.405622 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:29:25.405726 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260925 15:29:25.406175 22829 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 15:29:25.406317 22829 leader_election.cc:302] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0ec26006c04debb19f030d1a46fdb5; no voters:
I20260925 15:29:25.406522 22829 leader_election.cc:288] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:25.406605 22834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:25.406791 22834 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [term 1 LEADER]: Becoming Leader. State: Replica: df0ec26006c04debb19f030d1a46fdb5, State: Running, Role: LEADER
I20260925 15:29:25.406837 22829 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:25.406955 22834 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 15:29:25.407060 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:25.407943 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0ec26006c04debb19f030d1a46fdb5. Latest consensus state: current_term: 1 leader_uuid: "df0ec26006c04debb19f030d1a46fdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260925 15:29:25.408030 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:25.408268 22846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:25.408916 22846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:25.408941 22835 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0ec26006c04debb19f030d1a46fdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0ec26006c04debb19f030d1a46fdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260925 15:29:25.409010 22835 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:25.410704 22846 catalog_manager.cc:1347] Generated new cluster ID: e37ac1c4c4094d5988e519ab6b46c4ca
I20260925 15:29:25.410755 22846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:25.411231 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:25.408763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:41805 (local address 127.0.0.1:41819)
0925 15:29:25.408799 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:25.408801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:25.409366 (+ 565us) server_negotiation.cc:407] Connection header received
0925 15:29:25.409448 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:25.409449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:25.409478 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:25.409541 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:25.410915 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:25.410916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:25.410916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:25.410924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:25.410942 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:25.410948 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:25.411029 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:25.411148 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:29:25.411164 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:25.437672 22846 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:25.437891 22846 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:25.443393 22846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0ec26006c04debb19f030d1a46fdb5: Generated new TSK 0
I20260925 15:29:25.443722 22846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:25.452854 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22761
2026-09-25T15:29:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:25 dist-test-slave-hvv5 krb5kdc[22860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:25 dist-test-slave-hvv5 krb5kdc[22860](info): set up 2 sockets
Sep 25 15:29:25 dist-test-slave-hvv5 krb5kdc[22860](info): commencing operation
krb5kdc: starting...
W20260925 15:29:26.540977 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:26 dist-test-slave-hvv5 krb5kdc[22860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:26.710357 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:26.809221 22873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:26.809521 22873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:26.809566 22873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:26.811525 22873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:26.811650 22873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:26.811692 22873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:26.811720 22873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:26.811771 22873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:26.814357 22873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:26.814751 22873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:26.815026 22873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:26.822155 22879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:26.822355 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:26.822366 22878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:26.823017 22873 server_base.cc:1034] running on GCE node
I20260925 15:29:26.823364 22873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:26.823745 22873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:26.825062 22873 hybrid_clock.cc:648] HybridClock initialized: now 1790350166825011 us; error 68 us; skew 500 ppm
Sep 25 15:29:26 dist-test-slave-hvv5 krb5kdc[22860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:26.828182 22873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:26.828826 22873 webserver.cc:460] Webserver started at http://127.0.0.1:39761/ using document root <none> and password file <none>
I20260925 15:29:26.829103 22873 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:26.829173 22873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:26.829334 22873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:26.830540 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6"
format_stamp: "Formatted at 2026-09-25 15:29:26 on dist-test-slave-hvv5"
server_key: "1ab13986257989bc06602815462c694f"
server_key_iv: "36bba89f8db889acdacf6502670f4f86"
server_key_version: "encryptionkey@0"
I20260925 15:29:26.830943 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6"
format_stamp: "Formatted at 2026-09-25 15:29:26 on dist-test-slave-hvv5"
server_key: "1ab13986257989bc06602815462c694f"
server_key_iv: "36bba89f8db889acdacf6502670f4f86"
server_key_version: "encryptionkey@0"
I20260925 15:29:26.833277 22873 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260925 15:29:26.834422 22888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:26.834779 22873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:26.834942 22873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6"
format_stamp: "Formatted at 2026-09-25 15:29:26 on dist-test-slave-hvv5"
server_key: "1ab13986257989bc06602815462c694f"
server_key_iv: "36bba89f8db889acdacf6502670f4f86"
server_key_version: "encryptionkey@0"
I20260925 15:29:26.835047 22873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:26.860222 22873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:26.863416 22873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:26.863641 22873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:26.869894 22873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38791
I20260925 15:29:26.869992 22940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38791 every 8 connection(s)
I20260925 15:29:26.870536 22873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:26.871528 22941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:26.874359 22941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6: Bootstrap starting.
I20260925 15:29:26.875041 22941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:26.875507 22941 log.cc:824] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:26.876439 22941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6: No bootstrap required, opened a new log
I20260925 15:29:26.877377 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22873
I20260925 15:29:26.877472 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:26.877693 1677 external_mini_cluster.cc:1428] Setting key 309b13ac0f53a3962c4a023f6c064365
I20260925 15:29:26.879051 22941 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } }
I20260925 15:29:26.879289 22941 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:26.879339 22941 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09ca0cb7a3d46ab99e2074d3b18f0b6, State: Initialized, Role: FOLLOWER
I20260925 15:29:26.879479 22941 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } }
I20260925 15:29:26.879581 22941 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:26.879619 22941 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:26.879665 22941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:26.880751 22941 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } }
I20260925 15:29:26.880887 22941 leader_election.cc:302] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09ca0cb7a3d46ab99e2074d3b18f0b6; no voters:
I20260925 15:29:26.881090 22941 leader_election.cc:288] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:26.881114 22946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:26.881291 22946 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [term 1 LEADER]: Becoming Leader. State: Replica: f09ca0cb7a3d46ab99e2074d3b18f0b6, State: Running, Role: LEADER
I20260925 15:29:26.881433 22946 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } }
I20260925 15:29:26.881497 22941 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:26.882261 22948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } } }
I20260925 15:29:26.882943 22948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:26.882654 22947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09ca0cb7a3d46ab99e2074d3b18f0b6. Latest consensus state: current_term: 1 leader_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09ca0cb7a3d46ab99e2074d3b18f0b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38791 } } }
I20260925 15:29:26.883237 22947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:26.883435 22959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:26.884202 22959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:29:26 dist-test-slave-hvv5 krb5kdc[22860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:29:26.886456 22959 catalog_manager.cc:1347] Generated new cluster ID: eb509bfb0f85437f952eaa194d1ec54d
I20260925 15:29:26.886512 22959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:26.891898 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:26.878459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:59359 (local address 127.0.0.1:38791)
0925 15:29:26.878739 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:26.878747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:26.878783 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:29:26.878891 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:26.878896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:26.878946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:26.879061 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:26.879919 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:26.881672 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:26.882706 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:26.883594 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:26.883745 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:26.887499 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:26.887505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:26.887506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:26.887523 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:26.889807 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:26.890273 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:26.890277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:26.890277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:26.890400 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:26.890651 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:26.890653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:26.890653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:26.891187 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:26.891474 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:26.891566 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:29:26.891679 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20260925 15:29:26.893110 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:26.904241 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:26.897000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:36685 (local address 127.0.0.1:38791)
0925 15:29:26.897050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:26.897053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:26.897267 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:29:26.897345 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:26.897346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:26.897358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:26.897433 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:26.898032 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:26.898550 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:26.899343 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:26.899562 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:26.899644 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:26.901386 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:26.901388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:26.901388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:26.901396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:26.902982 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:26.903383 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:26.903384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:26.903385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:26.903447 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:26.903683 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:26.903684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:26.903684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:26.903808 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:26.903936 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:26.904058 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:29:26.904127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:29:26.917472 22959 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:26.918071 22959 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:26.924664 22959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09ca0cb7a3d46ab99e2074d3b18f0b6: Generated new TSK 0
I20260925 15:29:26.925011 22959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:26.931535 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22873
W20260925 15:29:26.937659 22858 connection.cc:535] client connection to 127.0.0.1:38791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38791): Connection reset by peer (error 104)
2026-09-25T15:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:27 dist-test-slave-hvv5 krb5kdc[22972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:27 dist-test-slave-hvv5 krb5kdc[22972](info): set up 2 sockets
Sep 25 15:29:27 dist-test-slave-hvv5 krb5kdc[22972](info): commencing operation
krb5kdc: starting...
W20260925 15:29:28.013554 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:28 dist-test-slave-hvv5 krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:28.157006 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:28.249732 22985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:28.250008 22985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:28.250061 22985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:28.251364 22985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:28.251448 22985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:28.251473 22985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:28.251495 22985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:28.251521 22985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:28.253211 22985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:28.253492 22985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:28.253686 22985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:28.259974 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:28.259979 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:28.260039 22985 server_base.cc:1034] running on GCE node
W20260925 15:29:28.259979 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:28.260526 22985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:28.260947 22985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:28.262138 22985 hybrid_clock.cc:648] HybridClock initialized: now 1790350168262127 us; error 31 us; skew 500 ppm
Sep 25 15:29:28 dist-test-slave-hvv5 krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:28.265143 22985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:28.265668 22985 webserver.cc:460] Webserver started at http://127.0.0.1:41399/ using document root <none> and password file <none>
I20260925 15:29:28.265929 22985 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:28.266001 22985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:28.266145 22985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:28.267278 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "c6c093a32afc46e0880cbeff4f5a6a7a"
format_stamp: "Formatted at 2026-09-25 15:29:28 on dist-test-slave-hvv5"
server_key: "e7192cbc3583829d88d24cbcb11c4516"
server_key_iv: "0bb4e6b1d431a9ea171afdc1d0a752d2"
server_key_version: "encryptionkey@0"
I20260925 15:29:28.267642 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "c6c093a32afc46e0880cbeff4f5a6a7a"
format_stamp: "Formatted at 2026-09-25 15:29:28 on dist-test-slave-hvv5"
server_key: "e7192cbc3583829d88d24cbcb11c4516"
server_key_iv: "0bb4e6b1d431a9ea171afdc1d0a752d2"
server_key_version: "encryptionkey@0"
I20260925 15:29:28.269187 22985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:28.270044 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:28.270263 22985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:28.270368 22985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "c6c093a32afc46e0880cbeff4f5a6a7a"
format_stamp: "Formatted at 2026-09-25 15:29:28 on dist-test-slave-hvv5"
server_key: "e7192cbc3583829d88d24cbcb11c4516"
server_key_iv: "0bb4e6b1d431a9ea171afdc1d0a752d2"
server_key_version: "encryptionkey@0"
I20260925 15:29:28.270443 22985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:28.288395 22985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:28.296346 22985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:28.296541 22985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:28.302168 22985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43927
I20260925 15:29:28.302822 22985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:28.304219 23053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:28.304687 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43927 every 8 connection(s)
I20260925 15:29:28.304800 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 22985
I20260925 15:29:28.304919 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:28.305094 1677 external_mini_cluster.cc:1428] Setting key cd3306961fa9a8b7a2f866969b366f3c
I20260925 15:29:28.308959 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a: Bootstrap starting.
I20260925 15:29:28.309808 23053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:28.310226 23053 log.cc:824] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:28.311182 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a: No bootstrap required, opened a new log
I20260925 15:29:28.312624 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:28.305690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:55407 (local address 127.0.0.1:43927)
0925 15:29:28.305910 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:28.305921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:28.305960 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:29:28.306718 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:28.306729 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:28.306813 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:28.307004 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:28.307274 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:28.308691 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:28.309851 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:28.310488 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:28.310606 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:28.311730 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:28.311735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:28.311735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:28.311967 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:28.312018 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:28.312033 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:28.312110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:28.312325 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:29:28.312360 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260925 15:29:28.313795 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:28.314798 23053 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } }
I20260925 15:29:28.315092 23053 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:28.315203 23053 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c093a32afc46e0880cbeff4f5a6a7a, State: Initialized, Role: FOLLOWER
I20260925 15:29:28.315442 23053 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } }
I20260925 15:29:28.315557 23053 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:28.315670 23053 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:28.315771 23053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:28.316920 23053 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } }
I20260925 15:29:28.317113 23053 leader_election.cc:302] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6c093a32afc46e0880cbeff4f5a6a7a; no voters:
I20260925 15:29:28.317402 23053 leader_election.cc:288] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:28.317449 23063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:28.317669 23063 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [term 1 LEADER]: Becoming Leader. State: Replica: c6c093a32afc46e0880cbeff4f5a6a7a, State: Running, Role: LEADER
I20260925 15:29:28.317735 23053 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:28.317821 23063 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } }
I20260925 15:29:28.318265 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c093a32afc46e0880cbeff4f5a6a7a. Latest consensus state: current_term: 1 leader_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } } }
I20260925 15:29:28.318480 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:28.318414 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c093a32afc46e0880cbeff4f5a6a7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43927 } } }
I20260925 15:29:28.318739 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:28.319342 23072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:28.320225 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:28.315653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:56129 (local address 127.0.0.1:43927)
0925 15:29:28.315701 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:28.315703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:28.315862 (+ 159us) server_negotiation.cc:407] Connection header received
0925 15:29:28.315967 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:28.315968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:28.315979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:28.316009 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:28.316218 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:28.317107 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:28.318226 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:28.318568 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:28.318625 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:28.319737 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:28.319739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:28.319739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:28.319744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:28.319759 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:28.319763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:28.319798 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:28.320094 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:29:28.320110 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:29:28.320230 23072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:28.325233 23072 catalog_manager.cc:1347] Generated new cluster ID: 4548237f3c264e32a0149ad86e08875c
I20260925 15:29:28.325326 23072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:28.334000 23072 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:28.334571 23072 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:28.337632 23072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c093a32afc46e0880cbeff4f5a6a7a: Generated new TSK 0
I20260925 15:29:28.337872 23072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:28.343086 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 22985
W20260925 15:29:28.348134 22970 connection.cc:535] client connection to 127.0.0.1:43927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43927): Connection reset by peer (error 104)
2026-09-25T15:29:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:28 dist-test-slave-hvv5 krb5kdc[23084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:28 dist-test-slave-hvv5 krb5kdc[23084](info): set up 2 sockets
Sep 25 15:29:28 dist-test-slave-hvv5 krb5kdc[23084](info): commencing operation
krb5kdc: starting...
W20260925 15:29:29.428979 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:29 dist-test-slave-hvv5 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:29:29.560993 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:29.651477 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:29.651764 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:29.651822 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:29.653209 23097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:29.653294 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:29.653317 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:29.653335 23097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:29.653357 23097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:29.655043 23097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:29.655362 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:29.655616 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:29.660923 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:29.661068 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:29.660938 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:29.661283 23097 server_base.cc:1034] running on GCE node
I20260925 15:29:29.661531 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:29.661935 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:29.663115 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790350169663103 us; error 50 us; skew 500 ppm
Sep 25 15:29:29 dist-test-slave-hvv5 krb5kdc[23084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:29.666040 23097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:29:29.666605 23097 webserver.cc:460] Webserver started at http://127.0.0.1:42049/ using document root <none> and password file <none>
I20260925 15:29:29.666847 23097 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:29.666916 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:29.667273 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:29.668433 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "61906d7d4865425b992984124a1da844"
format_stamp: "Formatted at 2026-09-25 15:29:29 on dist-test-slave-hvv5"
server_key: "f2b065191a7c031521de0aee93e5a745"
server_key_iv: "82371162407e54894f376d974ad4b137"
server_key_version: "encryptionkey@0"
I20260925 15:29:29.668876 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "61906d7d4865425b992984124a1da844"
format_stamp: "Formatted at 2026-09-25 15:29:29 on dist-test-slave-hvv5"
server_key: "f2b065191a7c031521de0aee93e5a745"
server_key_iv: "82371162407e54894f376d974ad4b137"
server_key_version: "encryptionkey@0"
I20260925 15:29:29.670356 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:29.671195 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:29.671373 23097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:29:29.671510 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "61906d7d4865425b992984124a1da844"
format_stamp: "Formatted at 2026-09-25 15:29:29 on dist-test-slave-hvv5"
server_key: "f2b065191a7c031521de0aee93e5a745"
server_key_iv: "82371162407e54894f376d974ad4b137"
server_key_version: "encryptionkey@0"
I20260925 15:29:29.671588 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:29.675155 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:29.677578 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:29.677747 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:29.682170 23097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39641
I20260925 15:29:29.682202 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39641 every 8 connection(s)
I20260925 15:29:29.682562 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:29.683599 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:29.686204 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844: Bootstrap starting.
I20260925 15:29:29.686658 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23097
I20260925 15:29:29.686791 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:29.686986 1677 external_mini_cluster.cc:1428] Setting key d89a4f333056293f0bf420c4b9cf8d6f
I20260925 15:29:29.687026 23165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:29.687376 23165 log.cc:824] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:29.688333 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844: No bootstrap required, opened a new log
I20260925 15:29:29.690171 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:29.687546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:54151 (local address 127.0.0.1:39641)
0925 15:29:29.687767 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:29.687778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:29.688021 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:29:29.688681 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:29.688690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:29.688729 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:29.688888 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:29.689719 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:29.689728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:29.689728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:29.689733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:29.689773 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:29.689783 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:29.689869 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:29.689964 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:29:29.690018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260925 15:29:29.690783 23165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260925 15:29:29.690987 23165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:29.691042 23165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61906d7d4865425b992984124a1da844, State: Initialized, Role: FOLLOWER
I20260925 15:29:29.691162 23165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260925 15:29:29.691262 23165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:29.691293 23165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:29.691377 23165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:29.691394 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:29.692232 23165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260925 15:29:29.692358 23165 leader_election.cc:302] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61906d7d4865425b992984124a1da844; no voters:
I20260925 15:29:29.692559 23165 leader_election.cc:288] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:29.692607 23171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:29.692795 23171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [term 1 LEADER]: Becoming Leader. State: Replica: 61906d7d4865425b992984124a1da844, State: Running, Role: LEADER
I20260925 15:29:29.692837 23165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:29.692924 23171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260925 15:29:29.693471 23171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61906d7d4865425b992984124a1da844. Latest consensus state: current_term: 1 leader_uuid: "61906d7d4865425b992984124a1da844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260925 15:29:29.693482 23176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61906d7d4865425b992984124a1da844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61906d7d4865425b992984124a1da844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260925 15:29:29.693606 23171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:29.693682 23176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:29.694031 23184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:29.694886 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:29.693001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.177:51761 (local address 127.0.0.1:39641)
0925 15:29:29.693045 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:29.693047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:29.693187 (+ 140us) server_negotiation.cc:407] Connection header received
0925 15:29:29.693284 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:29.693285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:29.693406 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:29.693459 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:29.694513 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:29.694515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:29.694515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:29.694520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:29.694537 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:29.694541 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:29.694574 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:29.694698 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:29:29.694800 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:29:29.694895 23184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:29.696916 23184 catalog_manager.cc:1347] Generated new cluster ID: 3f5a498d04194e6a9096ce624de1b8f8
I20260925 15:29:29.696995 23184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:29.721803 23184 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:29.722033 23184 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:29.726488 23184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61906d7d4865425b992984124a1da844: Generated new TSK 0
I20260925 15:29:29.726709 23184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:29.735430 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23097
2026-09-25T15:29:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:29 dist-test-slave-hvv5 krb5kdc[23196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:29 dist-test-slave-hvv5 krb5kdc[23196](info): set up 2 sockets
Sep 25 15:29:29 dist-test-slave-hvv5 krb5kdc[23196](info): commencing operation
krb5kdc: starting...
W20260925 15:29:30.797580 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:30 dist-test-slave-hvv5 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:30.950223 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43347
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:31.042850 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:31.043107 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:31.043134 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:31.044612 23209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:31.044687 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:31.044703 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:31.044716 23209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:31.044729 23209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:31.046790 23209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:43347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:31.047111 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:31.047376 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:31.053512 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:31.053516 23209 server_base.cc:1034] running on GCE node
W20260925 15:29:31.053519 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:31.053928 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:31.054656 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:31.055114 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:31.056331 23209 hybrid_clock.cc:648] HybridClock initialized: now 1790350171056295 us; error 55 us; skew 500 ppm
Sep 25 15:29:31 dist-test-slave-hvv5 krb5kdc[23196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350171, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:31.067593 23209 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:31.068270 23209 webserver.cc:460] Webserver started at http://127.1.163.126:42113/ using document root <none> and password file <none>
I20260925 15:29:31.068616 23209 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:31.068701 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:31.068863 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:31.070243 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "5f0963ad403a42a8ba35a6ef21b2f15d"
format_stamp: "Formatted at 2026-09-25 15:29:31 on dist-test-slave-hvv5"
server_key: "a4461b0194478189d40b44609b3cde24"
server_key_iv: "30e0761d5817b8bdef2185c7afe28301"
server_key_version: "encryptionkey@0"
I20260925 15:29:31.070691 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "5f0963ad403a42a8ba35a6ef21b2f15d"
format_stamp: "Formatted at 2026-09-25 15:29:31 on dist-test-slave-hvv5"
server_key: "a4461b0194478189d40b44609b3cde24"
server_key_iv: "30e0761d5817b8bdef2185c7afe28301"
server_key_version: "encryptionkey@0"
I20260925 15:29:31.072247 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:31.073437 23224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:31.073724 23209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:31.073870 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "5f0963ad403a42a8ba35a6ef21b2f15d"
format_stamp: "Formatted at 2026-09-25 15:29:31 on dist-test-slave-hvv5"
server_key: "a4461b0194478189d40b44609b3cde24"
server_key_iv: "30e0761d5817b8bdef2185c7afe28301"
server_key_version: "encryptionkey@0"
I20260925 15:29:31.073983 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:31.092778 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:31.095872 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:31.096076 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:31.100842 23209 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:43347
I20260925 15:29:31.100926 23276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:43347 every 8 connection(s)
I20260925 15:29:31.101306 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:31.102445 23277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:31.105675 23277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d: Bootstrap starting.
I20260925 15:29:31.106529 23277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:31.106603 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23209
I20260925 15:29:31.106724 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:31.107089 23277 log.cc:824] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:31.107240 1677 external_mini_cluster.cc:1428] Setting key 8e6c312bbe6daba3fe216e4ab116f40e
I20260925 15:29:31.108224 23277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d: No bootstrap required, opened a new log
I20260925 15:29:31.110921 23277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } }
I20260925 15:29:31.111104 23277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:31.111156 23277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0963ad403a42a8ba35a6ef21b2f15d, State: Initialized, Role: FOLLOWER
I20260925 15:29:31.111307 23277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } }
I20260925 15:29:31.111405 23277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:31.111455 23277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:31.111515 23277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:31.112322 23277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } }
I20260925 15:29:31.112450 23277 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0963ad403a42a8ba35a6ef21b2f15d; no voters:
I20260925 15:29:31.112660 23277 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:31.112758 23282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:31.112967 23277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:31.113003 23282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [term 1 LEADER]: Becoming Leader. State: Replica: 5f0963ad403a42a8ba35a6ef21b2f15d, State: Running, Role: LEADER
I20260925 15:29:31.113124 23282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } }
I20260925 15:29:31.114303 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0963ad403a42a8ba35a6ef21b2f15d. Latest consensus state: current_term: 1 leader_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } } }
I20260925 15:29:31.114404 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [sys.catalog]: This master's current role is: LEADER
Sep 25 15:29:31 dist-test-slave-hvv5 krb5kdc[23196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:31.114866 23292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:31.115288 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0963ad403a42a8ba35a6ef21b2f15d" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 43347 } } }
I20260925 15:29:31.115402 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:31.115804 23292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:31.117602 23292 catalog_manager.cc:1347] Generated new cluster ID: 32b727e71ea4413ab955c9903be2f948
I20260925 15:29:31.117672 23292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:31.120328 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:31.108102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37835 (local address 127.1.163.126:43347)
0925 15:29:31.108300 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:31.108311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:31.108349 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:29:31.108860 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:31.108867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:31.108924 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:31.109080 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:31.109735 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:31.110630 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:31.111508 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:31.112166 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:31.112300 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:31.115677 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:31.115683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:31.115683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:31.115699 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:31.118170 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:31.118694 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:31.118697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:31.118698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:31.118818 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:31.119057 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:31.119058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:31.119059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:31.119402 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:31.119587 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:31.119687 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:29:31.119809 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260925 15:29:31.121411 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:31.131588 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:31.122944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49505 (local address 127.1.163.126:43347)
0925 15:29:31.122980 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:31.122982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:31.123118 (+ 136us) server_negotiation.cc:407] Connection header received
0925 15:29:31.123178 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:31.123179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:31.123190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:31.123260 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:31.123891 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:31.124718 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:31.126021 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:31.126360 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:31.126451 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:31.127997 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:31.127999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:31.128000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:31.128008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:31.130187 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:31.130661 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:31.130662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:31.130662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:31.130741 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:31.130980 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:31.130981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:31.130981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:31.131131 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:31.131244 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:31.131392 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:29:31.131465 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:31.133598 23292 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:31.134353 23292 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:31.143019 23292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0963ad403a42a8ba35a6ef21b2f15d: Generated new TSK 0
I20260925 15:29:31.143294 23292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:31.150020 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23209
W20260925 15:29:31.154784 23194 connection.cc:535] client connection to 127.1.163.126:43347 recv error: Network error: failed to read from TLS socket (remote: 127.1.163.126:43347): Connection reset by peer (error 104)
2026-09-25T15:29:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:31 dist-test-slave-hvv5 krb5kdc[23308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:31 dist-test-slave-hvv5 krb5kdc[23308](info): set up 2 sockets
Sep 25 15:29:31 dist-test-slave-hvv5 krb5kdc[23308](info): commencing operation
krb5kdc: starting...
W20260925 15:29:32.265202 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:32 dist-test-slave-hvv5 krb5kdc[23308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:32.417083 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:42189
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:32.505089 23321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:32.505342 23321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:32.505368 23321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:32.506688 23321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:32.506742 23321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:32.506755 23321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:32.506768 23321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:32.506783 23321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:32.508431 23321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:42189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:32.508739 23321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:32.508956 23321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:32.514393 23326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:32.514437 23329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:32.514482 23321 server_base.cc:1034] running on GCE node
W20260925 15:29:32.514441 23327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:32.515060 23321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:32.515437 23321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:32.516634 23321 hybrid_clock.cc:648] HybridClock initialized: now 1790350172516571 us; error 86 us; skew 500 ppm
Sep 25 15:29:32 dist-test-slave-hvv5 krb5kdc[23308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350172, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:32.519475 23321 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:32.519971 23321 webserver.cc:460] Webserver started at http://127.1.163.126:41405/ using document root <none> and password file <none>
I20260925 15:29:32.520219 23321 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:32.520295 23321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:32.520447 23321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:32.521615 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "154f4a980e7047afbe3014e85984e435"
format_stamp: "Formatted at 2026-09-25 15:29:32 on dist-test-slave-hvv5"
server_key: "8189585b571aa7145c4ceb7a66058ebc"
server_key_iv: "b5fa20e456cd33b62b67e255fc17adfa"
server_key_version: "encryptionkey@0"
I20260925 15:29:32.521999 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "154f4a980e7047afbe3014e85984e435"
format_stamp: "Formatted at 2026-09-25 15:29:32 on dist-test-slave-hvv5"
server_key: "8189585b571aa7145c4ceb7a66058ebc"
server_key_iv: "b5fa20e456cd33b62b67e255fc17adfa"
server_key_version: "encryptionkey@0"
I20260925 15:29:32.523711 23321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:32.524650 23336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:32.524848 23321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:32.524957 23321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "154f4a980e7047afbe3014e85984e435"
format_stamp: "Formatted at 2026-09-25 15:29:32 on dist-test-slave-hvv5"
server_key: "8189585b571aa7145c4ceb7a66058ebc"
server_key_iv: "b5fa20e456cd33b62b67e255fc17adfa"
server_key_version: "encryptionkey@0"
I20260925 15:29:32.525190 23321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:32.538028 23321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:32.540772 23321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:32.540982 23321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:32.545934 23321 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:42189
I20260925 15:29:32.546061 23388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:42189 every 8 connection(s)
I20260925 15:29:32.546413 23321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:32.547348 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:32.549809 23389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435: Bootstrap starting.
I20260925 15:29:32.550428 23389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:32.550714 23389 log.cc:824] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:32.551576 23389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435: No bootstrap required, opened a new log
I20260925 15:29:32.552902 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23321
I20260925 15:29:32.553057 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:32.553262 1677 external_mini_cluster.cc:1428] Setting key aba372717d308d3e7666c1504c2fa496
I20260925 15:29:32.554037 23389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } }
I20260925 15:29:32.554275 23389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:32.554311 23389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154f4a980e7047afbe3014e85984e435, State: Initialized, Role: FOLLOWER
I20260925 15:29:32.554440 23389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } }
I20260925 15:29:32.554535 23389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:32.554570 23389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:32.554673 23389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:32.555693 23389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } }
I20260925 15:29:32.555819 23389 leader_election.cc:302] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 154f4a980e7047afbe3014e85984e435; no voters:
I20260925 15:29:32.555989 23389 leader_election.cc:288] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:32.556061 23394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:32.556262 23389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:32.556313 23394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [term 1 LEADER]: Becoming Leader. State: Replica: 154f4a980e7047afbe3014e85984e435, State: Running, Role: LEADER
I20260925 15:29:32.556437 23394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } }
I20260925 15:29:32.556891 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "154f4a980e7047afbe3014e85984e435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } } }
I20260925 15:29:32.556983 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:32.557137 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 154f4a980e7047afbe3014e85984e435. Latest consensus state: current_term: 1 leader_uuid: "154f4a980e7047afbe3014e85984e435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154f4a980e7047afbe3014e85984e435" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42189 } } }
I20260925 15:29:32.557184 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:32.558122 23398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:32.558671 23398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:32.560943 23398 catalog_manager.cc:1347] Generated new cluster ID: 3d6a589699914102b1eac15ce8f20e11
I20260925 15:29:32.561044 23398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:32.561275 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:32.553891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37373 (local address 127.1.163.126:42189)
0925 15:29:32.554149 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:32.554160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:32.554194 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:29:32.554328 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:32.554336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:32.554397 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:32.554557 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:32.555215 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:32.556624 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:32.557904 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:32.558549 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:32.558663 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:32.560403 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:32.560408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:32.560408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:32.560658 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:32.560701 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:32.560718 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:32.560847 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:32.561014 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 15:29:32.561051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":140,"threads_started":1}
I20260925 15:29:32.562361 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:32.567939 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:32.563810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57505 (local address 127.1.163.126:42189)
0925 15:29:32.563854 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:32.563856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:32.563947 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:29:32.564050 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:32.564050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:32.564062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:32.564092 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:32.564305 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:32.565107 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:32.566293 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:32.566643 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:32.566692 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:32.567620 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:32.567621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:32.567622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:32.567627 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:32.567643 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:32.567647 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:32.567682 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:32.567835 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 15:29:32.567851 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:29:32.570628 23398 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:32.571228 23398 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:32.576884 23398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 154f4a980e7047afbe3014e85984e435: Generated new TSK 0
I20260925 15:29:32.577133 23398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:32.584762 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23321
W20260925 15:29:32.589561 23306 connection.cc:535] client connection to 127.1.163.126:42189 recv error: Network error: failed to read from TLS socket (remote: 127.1.163.126:42189): Connection reset by peer (error 104)
2026-09-25T15:29:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:32 dist-test-slave-hvv5 krb5kdc[23420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:32 dist-test-slave-hvv5 krb5kdc[23420](info): set up 2 sockets
Sep 25 15:29:32 dist-test-slave-hvv5 krb5kdc[23420](info): commencing operation
krb5kdc: starting...
W20260925 15:29:33.669304 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:33 dist-test-slave-hvv5 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:33.820586 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44869
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:44869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:33.913388 23433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:33.913643 23433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:33.913672 23433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:33.914978 23433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:33.915025 23433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:33.915040 23433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:33.915052 23433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:33.915066 23433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:33.916878 23433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:44869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:33.917143 23433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:33.917392 23433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:33.923316 23438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:33.923316 23439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:33.923355 23441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:33.923851 23433 server_base.cc:1034] running on GCE node
I20260925 15:29:33.924364 23433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:33.924909 23433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:33.926106 23433 hybrid_clock.cc:648] HybridClock initialized: now 1790350173926083 us; error 41 us; skew 500 ppm
Sep 25 15:29:33 dist-test-slave-hvv5 krb5kdc[23420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350173, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:33.929574 23433 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:33.930099 23433 webserver.cc:460] Webserver started at http://127.1.163.126:41455/ using document root <none> and password file <none>
I20260925 15:29:33.930356 23433 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:33.930426 23433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:33.930572 23433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:33.931723 23433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "859a9400a5b04ba19adffcd32e833837"
format_stamp: "Formatted at 2026-09-25 15:29:33 on dist-test-slave-hvv5"
server_key: "2de2930cc2699293f3b442ad36b0c41f"
server_key_iv: "edfd9b8ffd6334d363a8eec1e5c78b49"
server_key_version: "encryptionkey@0"
I20260925 15:29:33.932108 23433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "859a9400a5b04ba19adffcd32e833837"
format_stamp: "Formatted at 2026-09-25 15:29:33 on dist-test-slave-hvv5"
server_key: "2de2930cc2699293f3b442ad36b0c41f"
server_key_iv: "edfd9b8ffd6334d363a8eec1e5c78b49"
server_key_version: "encryptionkey@0"
I20260925 15:29:33.933715 23433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:33.934773 23448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:33.935066 23433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:33.935215 23433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "859a9400a5b04ba19adffcd32e833837"
format_stamp: "Formatted at 2026-09-25 15:29:33 on dist-test-slave-hvv5"
server_key: "2de2930cc2699293f3b442ad36b0c41f"
server_key_iv: "edfd9b8ffd6334d363a8eec1e5c78b49"
server_key_version: "encryptionkey@0"
I20260925 15:29:33.935307 23433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:33.939163 23433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:33.941704 23433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:33.941910 23433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:33.946619 23433 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:44869
I20260925 15:29:33.946666 23500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:44869 every 8 connection(s)
I20260925 15:29:33.947110 23433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:33.948177 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:33.950747 23501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837: Bootstrap starting.
I20260925 15:29:33.951325 23501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:33.951577 23501 log.cc:824] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:33.952791 23501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837: No bootstrap required, opened a new log
I20260925 15:29:33.954452 23501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } }
I20260925 15:29:33.954629 23501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:33.954654 23501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859a9400a5b04ba19adffcd32e833837, State: Initialized, Role: FOLLOWER
I20260925 15:29:33.954818 23501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } }
I20260925 15:29:33.954910 23501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:33.954938 23501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:33.954969 23501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:33.955932 23501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } }
I20260925 15:29:33.956080 23501 leader_election.cc:302] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 859a9400a5b04ba19adffcd32e833837; no voters:
I20260925 15:29:33.956125 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23433
I20260925 15:29:33.956219 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:33.956307 23501 leader_election.cc:288] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:33.956424 1677 external_mini_cluster.cc:1428] Setting key 07c8b926e843b8b9d99e68871c9aee35
I20260925 15:29:33.956632 23501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:33.956720 23505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:33.956934 23505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [term 1 LEADER]: Becoming Leader. State: Replica: 859a9400a5b04ba19adffcd32e833837, State: Running, Role: LEADER
I20260925 15:29:33.957069 23505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } }
I20260925 15:29:33.957479 23504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "859a9400a5b04ba19adffcd32e833837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } } }
I20260925 15:29:33.957616 23504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:33.957479 23506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 859a9400a5b04ba19adffcd32e833837. Latest consensus state: current_term: 1 leader_uuid: "859a9400a5b04ba19adffcd32e833837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859a9400a5b04ba19adffcd32e833837" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 44869 } } }
I20260925 15:29:33.957693 23506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:33.958220 23515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:33.958945 23515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:33.959692 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:33.957112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41037 (local address 127.1.163.126:44869)
0925 15:29:33.957468 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:33.957482 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:33.957522 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:29:33.957663 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:33.957671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:33.957967 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:33.958494 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:33.959263 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:33.959267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:33.959268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:33.959272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:33.959322 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:33.959332 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:33.959406 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:33.959478 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:29:33.959523 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":62,"threads_started":1}
I20260925 15:29:33.960708 23515 catalog_manager.cc:1347] Generated new cluster ID: 9cd0069fce974ce69f78ce704386dd35
I20260925 15:29:33.960765 23515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:33.960773 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:33.966506 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:33.964923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59747 (local address 127.1.163.126:44869)
0925 15:29:33.964963 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:33.964965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:33.965210 (+ 245us) server_negotiation.cc:407] Connection header received
0925 15:29:33.965289 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:33.965290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:33.965297 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:33.965365 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:33.966183 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:33.966185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:33.966185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:33.966191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:33.966211 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:33.966223 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:33.966284 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:33.966360 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:29:33.966375 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:33.976616 23515 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:33.976840 23515 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:33.982887 23515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 859a9400a5b04ba19adffcd32e833837: Generated new TSK 0
I20260925 15:29:33.983119 23515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:33.986896 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23433
2026-09-25T15:29:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:34 dist-test-slave-hvv5 krb5kdc[23532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:34 dist-test-slave-hvv5 krb5kdc[23532](info): set up 2 sockets
Sep 25 15:29:34 dist-test-slave-hvv5 krb5kdc[23532](info): commencing operation
krb5kdc: starting...
W20260925 15:29:35.089787 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:35.237478 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37155
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:33201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:35.329468 23545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:35.329710 23545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:35.329736 23545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:35.331091 23545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:35.331153 23545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:35.331169 23545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:35.331182 23545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:35.331202 23545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:35.332953 23545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:33201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:35.333222 23545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:35.333478 23545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:35.339351 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:35.339533 23545 server_base.cc:1034] running on GCE node
W20260925 15:29:35.339344 23551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:35.339344 23550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:35.339874 23545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:35.340201 23545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:35.341387 23545 hybrid_clock.cc:648] HybridClock initialized: now 1790350175341334 us; error 72 us; skew 500 ppm
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350175, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:35.344091 23545 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:35.344667 23545 webserver.cc:460] Webserver started at http://127.1.163.126:35157/ using document root <none> and password file <none>
I20260925 15:29:35.344950 23545 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:35.345022 23545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:35.345168 23545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:35.346282 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "013adf9102794477b9b373a891ca2423"
format_stamp: "Formatted at 2026-09-25 15:29:35 on dist-test-slave-hvv5"
server_key: "fb507ef622cbdcd728191df86e657e88"
server_key_iv: "362f241316a621badbc401b7b8499e83"
server_key_version: "encryptionkey@0"
I20260925 15:29:35.346686 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "013adf9102794477b9b373a891ca2423"
format_stamp: "Formatted at 2026-09-25 15:29:35 on dist-test-slave-hvv5"
server_key: "fb507ef622cbdcd728191df86e657e88"
server_key_iv: "362f241316a621badbc401b7b8499e83"
server_key_version: "encryptionkey@0"
I20260925 15:29:35.348199 23545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:35.349098 23560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:35.349289 23545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:29:35.349399 23545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "013adf9102794477b9b373a891ca2423"
format_stamp: "Formatted at 2026-09-25 15:29:35 on dist-test-slave-hvv5"
server_key: "fb507ef622cbdcd728191df86e657e88"
server_key_iv: "362f241316a621badbc401b7b8499e83"
server_key_version: "encryptionkey@0"
I20260925 15:29:35.349478 23545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:35.366173 23545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:35.368844 23545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:35.369022 23545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:35.373857 23545 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37155
I20260925 15:29:35.373891 23612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37155 every 8 connection(s)
I20260925 15:29:35.374270 23545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:35.375262 23613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:35.377753 23613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423: Bootstrap starting.
I20260925 15:29:35.378396 23613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:35.378705 23613 log.cc:824] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:35.379624 23613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423: No bootstrap required, opened a new log
I20260925 15:29:35.381491 23613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } }
I20260925 15:29:35.381657 23613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:35.381690 23613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013adf9102794477b9b373a891ca2423, State: Initialized, Role: FOLLOWER
I20260925 15:29:35.381847 23613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } }
I20260925 15:29:35.381927 23613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:35.381951 23613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:35.382002 23613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:35.382997 23613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } }
I20260925 15:29:35.383100 23613 leader_election.cc:302] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013adf9102794477b9b373a891ca2423; no voters:
I20260925 15:29:35.383322 23613 leader_election.cc:288] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:35.383350 23616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:35.383474 23616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [term 1 LEADER]: Becoming Leader. State: Replica: 013adf9102794477b9b373a891ca2423, State: Running, Role: LEADER
I20260925 15:29:35.383483 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23545
I20260925 15:29:35.383623 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:35.383641 23613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:35.383627 23616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } }
I20260925 15:29:35.383801 1677 external_mini_cluster.cc:1428] Setting key d17a54dc08e1f6fd023337d2444f54a2
I20260925 15:29:35.384158 23617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "013adf9102794477b9b373a891ca2423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } } }
I20260925 15:29:35.384270 23617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:35.384459 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013adf9102794477b9b373a891ca2423. Latest consensus state: current_term: 1 leader_uuid: "013adf9102794477b9b373a891ca2423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013adf9102794477b9b373a891ca2423" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37155 } } }
I20260925 15:29:35.384563 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:35.384802 23623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:35.385499 23623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:35.387419 23623 catalog_manager.cc:1347] Generated new cluster ID: 55bec7942cf44d3394ab3c657e31c34d
I20260925 15:29:35.387488 23623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:35.394791 23623 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:35.395521 23623 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:35.395788 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:35.384560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46239 (local address 127.1.163.126:37155)
0925 15:29:35.384848 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:35.384858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:35.384893 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:29:35.385059 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:35.385067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:35.385135 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:35.386413 (+ 1278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:35.386525 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:35.387881 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:35.388723 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:35.389171 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:35.389289 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:35.391705 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:35.391709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:35.391710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:35.391729 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:35.393817 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:35.394281 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:35.394285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:35.394285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:35.394422 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:35.394688 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:35.394689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:35.394689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:35.395024 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:35.395201 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:35.395306 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:29:35.395498 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
I20260925 15:29:35.396860 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:35.406687 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:35.398326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45955 (local address 127.1.163.126:37155)
0925 15:29:35.398363 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:35.398365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:35.398489 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:29:35.398552 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:35.398553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:35.398562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:35.398622 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:35.399296 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:35.400088 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:35.401242 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:35.401567 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:35.401639 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:35.403420 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:35.403422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:35.403423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:35.403431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:35.405310 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:35.405731 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:35.405733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:35.405734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:35.405875 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:35.406090 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:35.406091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:35.406092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:35.406252 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:35.406354 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:35.406486 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:29:35.406582 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:35.409256 23623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 013adf9102794477b9b373a891ca2423: Generated new TSK 0
I20260925 15:29:35.409443 23623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:35.413774 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23545
2026-09-25T15:29:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23644](info): set up 2 sockets
Sep 25 15:29:35 dist-test-slave-hvv5 krb5kdc[23644](info): commencing operation
krb5kdc: starting...
W20260925 15:29:36.517359 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:36 dist-test-slave-hvv5 krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:36.680347 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:42517
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:36.777652 23657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:36.777992 23657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:36.778039 23657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:36.779757 23657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:36.779886 23657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:36.779914 23657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:36.779937 23657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:36.779963 23657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:36.781996 23657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:36.782354 23657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:36.782615 23657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:36.789213 23663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:36.789315 23662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:36.789318 23665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:36.789768 23657 server_base.cc:1034] running on GCE node
I20260925 15:29:36.790078 23657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:36.790434 23657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:36.791623 23657 hybrid_clock.cc:648] HybridClock initialized: now 1790350176791591 us; error 54 us; skew 500 ppm
Sep 25 15:29:36 dist-test-slave-hvv5 krb5kdc[23644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:36.800312 23657 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:36.800931 23657 webserver.cc:460] Webserver started at http://127.1.163.126:46593/ using document root <none> and password file <none>
I20260925 15:29:36.801244 23657 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:36.801304 23657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:36.801445 23657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:36.802645 23657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "981c8ff3608e4286a9189b5ec3769d38"
format_stamp: "Formatted at 2026-09-25 15:29:36 on dist-test-slave-hvv5"
server_key: "e7ca343e86e3e913a273b44784d62983"
server_key_iv: "3851970f7099fa3eb98d2de225bb4d0a"
server_key_version: "encryptionkey@0"
I20260925 15:29:36.803061 23657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "981c8ff3608e4286a9189b5ec3769d38"
format_stamp: "Formatted at 2026-09-25 15:29:36 on dist-test-slave-hvv5"
server_key: "e7ca343e86e3e913a273b44784d62983"
server_key_iv: "3851970f7099fa3eb98d2de225bb4d0a"
server_key_version: "encryptionkey@0"
I20260925 15:29:36.804687 23657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:36.805656 23672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:36.805948 23657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:36.806063 23657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "981c8ff3608e4286a9189b5ec3769d38"
format_stamp: "Formatted at 2026-09-25 15:29:36 on dist-test-slave-hvv5"
server_key: "e7ca343e86e3e913a273b44784d62983"
server_key_iv: "3851970f7099fa3eb98d2de225bb4d0a"
server_key_version: "encryptionkey@0"
I20260925 15:29:36.806154 23657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:36.817119 23657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:36.819924 23657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:36.820137 23657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:36.826485 23657 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:42517
I20260925 15:29:36.826645 23724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:42517 every 8 connection(s)
I20260925 15:29:36.827006 23657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:36.828029 23725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:36.831121 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38: Bootstrap starting.
I20260925 15:29:36.832068 23725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:36.832429 23725 log.cc:824] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:36.833369 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38: No bootstrap required, opened a new log
I20260925 15:29:36.835462 23725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } }
I20260925 15:29:36.835675 23725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:36.835718 23725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981c8ff3608e4286a9189b5ec3769d38, State: Initialized, Role: FOLLOWER
I20260925 15:29:36.835920 23725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } }
I20260925 15:29:36.836018 23725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:36.836061 23725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:36.836114 23725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:36.837116 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23657
I20260925 15:29:36.837211 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:36.837251 23725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } }
I20260925 15:29:36.837443 1677 external_mini_cluster.cc:1428] Setting key cde01e14acc9c33988599e6daefc03a9
I20260925 15:29:36.837580 23725 leader_election.cc:302] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981c8ff3608e4286a9189b5ec3769d38; no voters:
I20260925 15:29:36.837992 23725 leader_election.cc:288] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:36.838121 23728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:36.838301 23728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [term 1 LEADER]: Becoming Leader. State: Replica: 981c8ff3608e4286a9189b5ec3769d38, State: Running, Role: LEADER
I20260925 15:29:36.838366 23725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:36.838454 23728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } }
I20260925 15:29:36.838888 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981c8ff3608e4286a9189b5ec3769d38. Latest consensus state: current_term: 1 leader_uuid: "981c8ff3608e4286a9189b5ec3769d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } } }
I20260925 15:29:36.838914 23730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981c8ff3608e4286a9189b5ec3769d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981c8ff3608e4286a9189b5ec3769d38" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 42517 } } }
I20260925 15:29:36.839021 23730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:36.839175 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:36.839263 23739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:36.839847 23739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:36.841745 23739 catalog_manager.cc:1347] Generated new cluster ID: 3e7ecc5244424ad8b5e259070a1ad93e
I20260925 15:29:36.841815 23739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:36.844506 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:36.838250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58017 (local address 127.1.163.126:42517)
0925 15:29:36.838642 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:36.838652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:36.838703 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:29:36.839304 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:36.839314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:36.839380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:36.840108 (+ 728us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:36.840224 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:36.841498 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:36.842408 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:36.842855 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:36.842998 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:36.843753 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:36.843758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:36.843758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:36.843950 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:36.843984 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:36.843996 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:36.844073 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:36.844151 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:29:36.844323 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":79,"threads_started":1}
I20260925 15:29:36.845477 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:36.852371 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:36.847354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36187 (local address 127.1.163.126:42517)
0925 15:29:36.847395 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:36.847397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:36.849030 (+ 1633us) server_negotiation.cc:407] Connection header received
0925 15:29:36.849120 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:36.849122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:36.849134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:36.849184 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:36.849334 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:36.849833 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:36.850548 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:36.850756 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:36.850829 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:36.852047 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:36.852048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:36.852049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:36.852054 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:36.852077 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:36.852081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:36.852136 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:36.852221 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:29:36.852259 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:29:36.855039 23739 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:36.855803 23739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:36.864346 23739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981c8ff3608e4286a9189b5ec3769d38: Generated new TSK 0
I20260925 15:29:36.864647 23739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:36.869244 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23657
2026-09-25T15:29:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:36 dist-test-slave-hvv5 krb5kdc[23756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:36 dist-test-slave-hvv5 krb5kdc[23756](info): set up 2 sockets
Sep 25 15:29:36 dist-test-slave-hvv5 krb5kdc[23756](info): commencing operation
krb5kdc: starting...
W20260925 15:29:37.993387 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:38 dist-test-slave-hvv5 krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:38.158457 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38793
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:38.256439 23769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:38.256709 23769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:38.256735 23769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:38.257959 23769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:38.258005 23769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:38.258020 23769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:38.258033 23769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:38.258045 23769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:38.259610 23769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:38.259850 23769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:38.260056 23769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:38.265926 23774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:38.265933 23777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:38.265959 23769 server_base.cc:1034] running on GCE node
W20260925 15:29:38.265933 23775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:38.266505 23769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:38.266894 23769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:38.268105 23769 hybrid_clock.cc:648] HybridClock initialized: now 1790350178268052 us; error 72 us; skew 500 ppm
Sep 25 15:29:38 dist-test-slave-hvv5 krb5kdc[23756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350178, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:38.271338 23769 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:38.271905 23769 webserver.cc:460] Webserver started at http://127.1.163.126:44209/ using document root <none> and password file <none>
I20260925 15:29:38.272156 23769 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:38.272235 23769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:38.272394 23769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:38.273818 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "f85cbb66601641f1bf489e88b8ccbac1"
format_stamp: "Formatted at 2026-09-25 15:29:38 on dist-test-slave-hvv5"
server_key: "3eee55ebf03f1ac3b8c9d7ef5f16f80f"
server_key_iv: "5b5d6e184fb60c58df24ee8f64217938"
server_key_version: "encryptionkey@0"
I20260925 15:29:38.274274 23769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "f85cbb66601641f1bf489e88b8ccbac1"
format_stamp: "Formatted at 2026-09-25 15:29:38 on dist-test-slave-hvv5"
server_key: "3eee55ebf03f1ac3b8c9d7ef5f16f80f"
server_key_iv: "5b5d6e184fb60c58df24ee8f64217938"
server_key_version: "encryptionkey@0"
I20260925 15:29:38.276466 23769 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:29:38.277448 23784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:38.277868 23769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:38.278026 23769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "f85cbb66601641f1bf489e88b8ccbac1"
format_stamp: "Formatted at 2026-09-25 15:29:38 on dist-test-slave-hvv5"
server_key: "3eee55ebf03f1ac3b8c9d7ef5f16f80f"
server_key_iv: "5b5d6e184fb60c58df24ee8f64217938"
server_key_version: "encryptionkey@0"
I20260925 15:29:38.278126 23769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:38.282238 23769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:38.284955 23769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:38.285137 23769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:38.290099 23769 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:38793
I20260925 15:29:38.290277 23836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:38793 every 8 connection(s)
I20260925 15:29:38.290580 23769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:38.291036 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23769
I20260925 15:29:38.291188 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:38.291441 1677 external_mini_cluster.cc:1428] Setting key 14c47fc1da1530e992e3fdc5753cd225
I20260925 15:29:38.291942 23837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:38.294726 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:38.292099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45745 (local address 127.1.163.126:38793)
0925 15:29:38.292294 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:38.292302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:38.292334 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:29:38.292871 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:38.292878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:38.292908 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:38.293074 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:38.293992 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:38.293995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:38.293996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:38.294000 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:38.294042 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:38.294054 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:38.294431 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:38.294511 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:29:38.294559 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260925 15:29:38.296075 23837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1: Bootstrap starting.
I20260925 15:29:38.296311 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:38.296919 23837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:38.297299 23837 log.cc:824] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:38.298549 23837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1: No bootstrap required, opened a new log
I20260925 15:29:38.299446 23840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:38.298210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36189 (local address 127.1.163.126:38793)
0925 15:29:38.298259 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:38.298261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:38.298398 (+ 137us) server_negotiation.cc:407] Connection header received
0925 15:29:38.298460 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:38.298461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:38.298467 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:38.298540 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:38.299194 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:38.299195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:38.299195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:29:38.299200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:38.299215 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:29:38.299219 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:38.299296 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:38.299357 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:29:38.299374 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:29:38.301077 23837 raft_consensus.cc:348] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } }
I20260925 15:29:38.301251 23837 raft_consensus.cc:374] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:29:38.301276 23837 raft_consensus.cc:729] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85cbb66601641f1bf489e88b8ccbac1, State: Initialized, Role: FOLLOWER
I20260925 15:29:38.301409 23837 consensus_queue.cc:260] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } }
I20260925 15:29:38.301494 23837 raft_consensus.cc:388] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:29:38.301518 23837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:29:38.301550 23837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:38.302208 23837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } }
I20260925 15:29:38.302306 23837 leader_election.cc:302] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f85cbb66601641f1bf489e88b8ccbac1; no voters:
I20260925 15:29:38.302484 23837 leader_election.cc:288] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:29:38.302523 23847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:38.302644 23847 raft_consensus.cc:686] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [term 1 LEADER]: Becoming Leader. State: Replica: f85cbb66601641f1bf489e88b8ccbac1, State: Running, Role: LEADER
I20260925 15:29:38.302752 23837 sys_catalog.cc:564] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:38.302799 23847 consensus_queue.cc:237] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } }
I20260925 15:29:38.303231 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f85cbb66601641f1bf489e88b8ccbac1. Latest consensus state: current_term: 1 leader_uuid: "f85cbb66601641f1bf489e88b8ccbac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } } }
I20260925 15:29:38.303422 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:38.303233 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f85cbb66601641f1bf489e88b8ccbac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85cbb66601641f1bf489e88b8ccbac1" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38793 } } }
I20260925 15:29:38.303748 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:38.303890 23856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:38.304669 23856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:38.306692 23856 catalog_manager.cc:1347] Generated new cluster ID: 16e7d9218549418e967d07dd97f1b552
I20260925 15:29:38.306769 23856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:38.320643 23856 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:38.320981 23856 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:38.336282 23856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f85cbb66601641f1bf489e88b8ccbac1: Generated new TSK 0
I20260925 15:29:38.336570 23856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:29:38.343081 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23769
2026-09-25T15:29:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1472 ms)
[----------] 18 tests from AuthTokenIssuingTest (25354 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:38 dist-test-slave-hvv5 krb5kdc[23868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:38 dist-test-slave-hvv5 krb5kdc[23868](info): set up 2 sockets
Sep 25 15:29:38 dist-test-slave-hvv5 krb5kdc[23868](info): commencing operation
krb5kdc: starting...
W20260925 15:29:39.453150 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:39.617672 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38337
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:39.709964 23881 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:39.710258 23881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:39.710290 23881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:39.710345 23881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:39.711633 23881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:39.711711 23881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:39.711735 23881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:39.711756 23881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:39.711795 23881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:39.713532 23881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:39.713862 23881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:39.714125 23881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:39.719774 23886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:39.719765 23887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:39.719765 23889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:39.720937 23881 server_base.cc:1034] running on GCE node
I20260925 15:29:39.721207 23881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:39.721540 23881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:39.722711 23881 hybrid_clock.cc:648] HybridClock initialized: now 1790350179722688 us; error 49 us; skew 500 ppm
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:39.725667 23881 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:39.726343 23881 webserver.cc:460] Webserver started at http://127.1.163.126:46749/ using document root <none> and password file <none>
I20260925 15:29:39.726723 23881 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:39.726805 23881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:39.726996 23881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:39.728133 23881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "325d6f983e864fc4b9266fc3226bf317"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "16da4271982a0a338e691a6118910f4a"
server_key_iv: "a35c62e99549bd702924f25e6d6eb106"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.728519 23881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "325d6f983e864fc4b9266fc3226bf317"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "16da4271982a0a338e691a6118910f4a"
server_key_iv: "a35c62e99549bd702924f25e6d6eb106"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.730151 23881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:39.731413 23896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:39.731714 23881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:39.731832 23881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "325d6f983e864fc4b9266fc3226bf317"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "16da4271982a0a338e691a6118910f4a"
server_key_iv: "a35c62e99549bd702924f25e6d6eb106"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.731909 23881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:39.751387 23881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:39.754043 23881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:39.754173 23881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:39.759552 23881 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:38337
I20260925 15:29:39.759590 23948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:38337 every 8 connection(s)
I20260925 15:29:39.759991 23881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:39.761168 23949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:39.762888 23949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:39.763962 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23881
I20260925 15:29:39.764102 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:39.764379 1677 external_mini_cluster.cc:1428] Setting key 3cf0685bb2002019a443304b32bb2560
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.125@KRBTEST.COM; defaulting to no policy
I20260925 15:29:39.777359 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.763878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.1.163.126:38337)
0925 15:29:39.764075 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:39.764081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:39.764121 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:29:39.765107 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:39.765109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:39.765137 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:39.765243 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:39.765988 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.767102 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.768809 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.769521 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.769648 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.772757 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:39.772767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:39.772768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:39.772778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:39.775121 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.775654 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.775657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.775657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.775870 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.776182 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.776183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.776184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.776525 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:39.776658 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:39.777104 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 15:29:39.777184 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":124,"threads_started":1}
I20260925 15:29:39.777366 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.763725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38337 (local address 127.0.0.1:51954)
0925 15:29:39.763972 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.764025 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:39.764910 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:39.765272 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:39.765274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:39.765588 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:39.765776 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.765781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.767242 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.767243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.768488 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.768491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.769759 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.769762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.769859 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.770583 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:39.770596 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:39.772647 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:39.775261 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.775269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.775271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.775483 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.775963 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.775964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.775964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.776057 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.776800 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:39.776807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:39.776876 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 15:29:39.776967 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:29:39.777113 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260925 15:29:39.778523 23949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:39.779151 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.778987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49682)
0925 15:29:39.779062 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.779118 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:29:39.779475 23900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
I20260925 15:29:39.780234 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.780066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49694)
0925 15:29:39.780166 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.780200 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.125@KRBTEST.COM" created.
W20260925 15:29:39.780490 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:38715: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
I20260925 15:29:39.800938 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:38715
--webserver_interface=127.1.163.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:29:39.830135 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } attempt: 1
I20260925 15:29:39.830844 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.830682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49708)
0925 15:29:39.830755 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.830805 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:29:39.831329 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.831220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49716)
0925 15:29:39.831269 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.831300 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:29:39.831497 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:38715: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
W20260925 15:29:39.897570 23956 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:39.898039 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:39.898304 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:39.898394 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 15:29:39.899710 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } attempt: 2
I20260925 15:29:39.900206 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.900037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49722)
0925 15:29:39.900116 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.900167 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:29:39.900297 23956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:39.900364 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:39.900388 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:39.900406 23956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:39.900431 23956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:39.900709 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.900574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49724)
0925 15:29:39.900639 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.900687 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:29:39.900851 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:38715: Network error: Client connection negotiation failed: client connection to 127.1.163.125:38715: connect: Connection refused (error 111)
I20260925 15:29:39.902741 23956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.163.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:39.903050 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:39.903254 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:39.909425 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:39.909963 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:39.910133 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:39.910557 23956 server_base.cc:1034] running on GCE node
I20260925 15:29:39.911041 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:39.911402 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:39.912720 23956 hybrid_clock.cc:648] HybridClock initialized: now 1790350179912692 us; error 42 us; skew 500 ppm
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:39.916275 23956 init.cc:377] Logged in from keytab as kudu/127.1.163.125@KRBTEST.COM (short username kudu)
I20260925 15:29:39.917061 23956 webserver.cc:460] Webserver started at http://127.1.163.125:45119/ using document root <none> and password file <none>
I20260925 15:29:39.917378 23956 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:39.917467 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:39.917640 23956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:39.918973 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data/instance:
uuid: "37a8a64415224c81a111814b3328408e"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "98dfafdb7b4c1488b39e1c40806286a3"
server_key_iv: "f766e4f5479322b264e177078e680e3b"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.919397 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal/instance:
uuid: "37a8a64415224c81a111814b3328408e"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "98dfafdb7b4c1488b39e1c40806286a3"
server_key_iv: "f766e4f5479322b264e177078e680e3b"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.921208 23956 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:29:39.922497 23971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:39.923156 23956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:29:39.923272 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "37a8a64415224c81a111814b3328408e"
format_stamp: "Formatted at 2026-09-25 15:29:39 on dist-test-slave-hvv5"
server_key: "98dfafdb7b4c1488b39e1c40806286a3"
server_key_iv: "f766e4f5479322b264e177078e680e3b"
server_key_version: "encryptionkey@0"
I20260925 15:29:39.923347 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:39.950811 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:39.953576 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:39.953715 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:39.957783 23956 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.125:38715
I20260925 15:29:39.957881 24023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.125:38715 every 8 connection(s)
I20260925 15:29:39.958251 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
I20260925 15:29:39.959796 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:39.961618 24024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:39.968393 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 23956
I20260925 15:29:39.968528 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-1/wal/instance
I20260925 15:29:39.968777 1677 external_mini_cluster.cc:1428] Setting key b2f585f151663ea299b4366aaa48ac89
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:39.975283 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.962721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.1.163.126:38337)
0925 15:29:39.962851 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:39.962854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:39.963187 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:29:39.963767 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:39.963768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:39.963779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:39.963923 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:39.965018 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.965678 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.967065 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.967284 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.967352 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.970765 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:39.970767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:39.970768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:39.970785 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:39.972310 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.973604 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.973605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.973605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.973686 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.974207 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.974209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.974209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.974351 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:39.974418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:39.974920 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 15:29:39.975207 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 15:29:39.975401 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.962745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38337 (local address 127.0.0.1:51964)
0925 15:29:39.963024 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.963064 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:39.963602 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:39.964078 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:39.964083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:39.964405 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:39.964699 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.964705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.965839 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.965841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.966795 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.966798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.967415 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.967416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.967752 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.968424 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:39.968436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:39.970635 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:39.972660 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.972672 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.972675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.973251 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.973769 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.973771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.973771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.973985 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.974428 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:39.974436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:39.974670 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 15:29:39.974825 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:29:39.975060 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":140,"threads_started":1}
I20260925 15:29:39.976102 24024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.163.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.124@KRBTEST.COM" created.
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:29:39.988402 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.976443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49740)
0925 15:29:39.976504 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.976518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:39.976797 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:39.976960 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:39.976962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:39.977129 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:39.977203 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.977205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.978269 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.978270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.979129 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.979132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.979749 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.979750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.979829 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.980531 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:39.980593 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:39.984150 (+ 3557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:39.986579 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.986581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.986581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.986885 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.987133 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:39.987134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:39.987135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:39.987209 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:39.987900 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:39.987901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:39.987960 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:29:39.988057 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:29:39.988195 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:29:39.988411 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.976482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.1.163.125:38715)
0925 15:29:39.976718 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:39.976722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:39.976781 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:29:39.976860 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:39.976862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:39.976874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:39.976936 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:39.977332 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.978169 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.979241 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.979565 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.979631 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.984287 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:39.984305 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:39.984306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:39.984316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:39.986478 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.986984 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.986987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.986988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.987055 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:39.987365 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:39.987367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:39.987367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:39.987732 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:39.987846 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:39.988117 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:29:39.988286 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":201,"thread_start_us":151,"threads_started":1}
I20260925 15:29:39.989473 24024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:39.989935 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.989803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45668)
0925 15:29:39.989859 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.989911 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:29:39.990088 23975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
I20260925 15:29:39.990578 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.990480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45674)
0925 15:29:39.990526 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.990558 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:29:39.990679 24024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
I20260925 15:29:39.992053 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } attempt: 3
Sep 25 15:29:39 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:29:40.001334 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.992466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.1.163.125:38715)
0925 15:29:39.992529 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:39.992531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:39.992573 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:29:39.992600 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:39.992601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:39.992615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:39.992690 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:39.993213 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.994091 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.995225 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.995516 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.995596 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.998202 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:39.998203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:39.998204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:39.998213 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:39.999991 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.000394 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.000396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.000397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.000462 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.000771 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.000772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.000772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.000902 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.000983 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.001139 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.001232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:29:40.001377 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:39.992379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49756)
0925 15:29:39.992438 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:39.992450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:39.992498 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:39.992766 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:39.992767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:39.993004 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:39.993073 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.993075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.994250 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.994251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.994913 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:39.994915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:39.995640 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:39.995641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:39.995789 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:39.996596 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:39.996602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:39.998079 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.000110 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.000112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.000113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.000308 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.000582 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.000583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.000583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.000671 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.000973 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.000974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.001035 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.001114 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.001281 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:29:40.002079 23949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:40.002494 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.002363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45678)
0925 15:29:40.002423 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.002468 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:29:40.003005 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.002868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45686)
0925 15:29:40.002950 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.002981 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 15:29:40.003203 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
I20260925 15:29:40.010465 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:40779
--webserver_interface=127.1.163.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:29:40.034440 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 1
I20260925 15:29:40.034986 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.034835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45702)
0925 15:29:40.034903 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.034951 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:29:40.035595 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.035411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45704)
0925 15:29:40.035536 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.035571 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260925 15:29:40.035786 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
I20260925 15:29:40.039954 24024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 1
I20260925 15:29:40.040491 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.040353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45720)
0925 15:29:40.040420 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.040461 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:29:40.041100 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.040985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45724)
0925 15:29:40.041038 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.041069 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:29:40.041222 24024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
W20260925 15:29:40.103015 24031 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:40.103245 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:40.103263 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:40.103284 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:40.104393 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:40.104434 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:40.104447 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:40.104460 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:40.104472 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:40.106053 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:38337,127.1.163.125:38715,127.1.163.124:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.163.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:40.106263 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:40.106457 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:29:40.109392 24024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 2
I20260925 15:29:40.109920 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.109757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45740)
0925 15:29:40.109821 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.109864 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:29:40.110517 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.110276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45744)
0925 15:29:40.110445 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.110486 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260925 15:29:40.110896 24024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
I20260925 15:29:40.110896 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 2
I20260925 15:29:40.111388 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.111235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45750)
0925 15:29:40.111325 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.111352 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:29:40.111776 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.111656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45764)
0925 15:29:40.111711 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.111743 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:29:40.111936 23949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:40779: Network error: Client connection negotiation failed: client connection to 127.1.163.124:40779: connect: Connection refused (error 111)
W20260925 15:29:40.112489 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:40.112478 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:40.112598 24031 server_base.cc:1034] running on GCE node
W20260925 15:29:40.112478 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:40.112936 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:40.113265 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:40.114490 24031 hybrid_clock.cc:648] HybridClock initialized: now 1790350180114440 us; error 69 us; skew 500 ppm
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:40.117430 24031 init.cc:377] Logged in from keytab as kudu/127.1.163.124@KRBTEST.COM (short username kudu)
I20260925 15:29:40.117985 24031 webserver.cc:460] Webserver started at http://127.1.163.124:34567/ using document root <none> and password file <none>
I20260925 15:29:40.118227 24031 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:40.118299 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:40.118459 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:40.119585 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data/instance:
uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf"
format_stamp: "Formatted at 2026-09-25 15:29:40 on dist-test-slave-hvv5"
server_key: "68d0a0b94d840e654b0d4596eb87aaa1"
server_key_iv: "d24993082e342dd19301a367d7af9524"
server_key_version: "encryptionkey@0"
I20260925 15:29:40.119983 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal/instance:
uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf"
format_stamp: "Formatted at 2026-09-25 15:29:40 on dist-test-slave-hvv5"
server_key: "68d0a0b94d840e654b0d4596eb87aaa1"
server_key_iv: "d24993082e342dd19301a367d7af9524"
server_key_version: "encryptionkey@0"
I20260925 15:29:40.121634 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:29:40.122604 24046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:40.122882 24031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:40.123008 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf"
format_stamp: "Formatted at 2026-09-25 15:29:40 on dist-test-slave-hvv5"
server_key: "68d0a0b94d840e654b0d4596eb87aaa1"
server_key_iv: "d24993082e342dd19301a367d7af9524"
server_key_version: "encryptionkey@0"
I20260925 15:29:40.123091 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:40.136065 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:40.138705 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:40.139200 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:40.143743 24031 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.124:40779
I20260925 15:29:40.143780 24098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.124:40779 every 8 connection(s)
I20260925 15:29:40.144177 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
I20260925 15:29:40.145329 24099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:40.146782 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 24031
I20260925 15:29:40.146893 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349904034090-1677-0/minicluster-data/master-2/wal/instance
I20260925 15:29:40.147111 1677 external_mini_cluster.cc:1428] Setting key 42fa8a9367ae244f61276fbcc1ad808b
I20260925 15:29:40.147442 24099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:40.159442 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.148596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.1.163.126:38337)
0925 15:29:40.148656 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.148658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.148781 (+ 123us) server_negotiation.cc:407] Connection header received
0925 15:29:40.148968 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.148969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.148990 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.149066 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.149664 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.150151 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.151062 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.151258 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.151373 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.155218 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.155221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.155221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.155231 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.158025 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158495 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.158496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.158496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.158557 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158786 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.158788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.158788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.158943 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.159007 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.159249 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.159344 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:29:40.159750 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.147897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.1.163.124:40779)
0925 15:29:40.148144 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.148153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.148179 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:29:40.148810 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.148820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.148872 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.148984 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.149652 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.150459 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.151214 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.151529 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.151618 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.155629 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.155638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.155638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.155646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.157889 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158394 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.158398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.158399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.158525 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158833 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.158834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.158834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.159109 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.159268 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.159367 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.159508 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260925 15:29:40.160301 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.148413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:38337 (local address 127.0.0.1:51974)
0925 15:29:40.148653 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.148703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:40.148869 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:40.149091 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:40.149093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:40.149382 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:40.149546 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.149550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.150240 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.150242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.150956 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.150958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.151401 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.151403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.151515 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.152230 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:40.152241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:40.155069 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.158115 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.158125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.158399 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.158631 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.158631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.158632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.158718 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.159033 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.159039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.159109 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.160053 (+ 944us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.160202 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":139,"threads_started":1}
I20260925 15:29:40.160995 24099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:40.171700 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.161142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.1.163.126:38337)
0925 15:29:40.161436 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.161437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.161446 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:40.161467 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.161467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.161477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.161535 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.162257 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.163019 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.163929 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.164136 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.164230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.167930 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.167932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.167932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.167941 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.170175 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.170776 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.170778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.170779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.170853 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.171081 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.171082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.171083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.171226 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.171424 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.171468 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.171564 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260925 15:29:40.171725 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.161456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:38715 (local address 127.0.0.1:49760)
0925 15:29:40.161582 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.161592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:40.161636 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:40.161827 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:40.161828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:40.161989 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:40.162049 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.162052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.163173 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.163174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.164111 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.164115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.164652 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.164653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.164734 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.165055 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:40.165061 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:40.167045 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.170523 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.170525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.170526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.170735 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.170995 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.170996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.170996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.171067 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.171348 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.171349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.171402 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.171502 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.171620 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 15:29:40.171725 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.161511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.1.163.125:38715)
0925 15:29:40.161579 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.161581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.161624 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:29:40.161695 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.161696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.161707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.161774 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.162148 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.163052 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.164211 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.164541 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.164642 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.167173 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.167175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.167176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.167186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.170379 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.170831 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.170833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.170834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.170905 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.171138 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.171139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.171140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.171269 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.171356 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.171529 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.171618 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:29:40.172384 24099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:29:40.182065 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.172734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.1.163.124:40779)
0925 15:29:40.172773 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.172775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.172812 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:29:40.172886 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.172887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.172903 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.172946 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.173302 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.174168 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.175333 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.175659 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.175781 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.178172 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.178174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.178174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.178181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.179830 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.180660 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.180661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.180662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.180736 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.181225 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.181228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.181228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.181395 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.181494 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.181853 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.181987 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":18}
I20260925 15:29:40.182194 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.172697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45772)
0925 15:29:40.172768 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.172780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:40.172826 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:40.172952 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:40.172953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:40.173154 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:40.173212 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.173214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.174258 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.174259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.175191 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.175194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.175794 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.175795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.175848 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.176528 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:40.176572 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:40.178060 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.180012 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.180014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.180014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.180465 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.180988 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.180989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.180989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.181062 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.181636 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.181638 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.181693 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.181783 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.181906 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:29:40.182799 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.172475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.1.163.125:38715)
0925 15:29:40.172525 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.172527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.172613 (+ 86us) server_negotiation.cc:407] Connection header received
0925 15:29:40.172719 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.172721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.172733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.172802 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.173495 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.174252 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.175339 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.175658 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.175743 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.178872 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.178874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.178874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.178882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.181045 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.181637 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.181638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.181638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.181708 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.182161 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.182162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.182162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.182311 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.182552 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.182592 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.182679 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:29:40.184448 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: Bootstrap starting.
I20260925 15:29:40.184777 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:40.185727 24099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:40.186082 24099 log.cc:824] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:40.185772 23923 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "325d6f983e864fc4b9266fc3226bf317"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51988
I20260925 15:29:40.187084 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: No bootstrap required, opened a new log
I20260925 15:29:40.188961 24099 raft_consensus.cc:348] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.189155 24099 raft_consensus.cc:729] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df53d04f3bf34dd9ab6d9d165bcb56cf, State: Initialized, Role: FOLLOWER
I20260925 15:29:40.189262 24099 consensus_queue.cc:260] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.189601 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.189697 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.189704 24099 sys_catalog.cc:564] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:29:40.192328 24116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:40.192402 24116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:40.193440 24031 master_runner.cc:186] Error getting master registration for 127.1.163.126:38337: OK, instance_id { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" instance_seqno: 1790350179757900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:29:40.196983 24031 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:29:40.202077 24024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 3
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:40.211047 23949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } attempt: 3
I20260925 15:29:40.212080 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.202613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.1.163.124:40779)
0925 15:29:40.202663 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.202665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.202677 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:29:40.202754 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.202755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.202767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.202846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.203280 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.204020 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.205422 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.205734 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.205810 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.208494 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.208496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.208496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.208505 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.210342 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.210829 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.210830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.210831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.210896 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.211164 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.211165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.211166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.211410 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.211613 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.211872 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.211977 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:29:40.212107 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.202534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45784)
0925 15:29:40.202609 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.202621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:40.202670 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:40.202868 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:40.202869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:40.203085 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:40.203151 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.203153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.204164 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.204166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.205306 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.205311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.205846 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.205847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.205917 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.206666 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:40.206672 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:40.208353 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.210454 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.210456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.210456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.210732 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.210985 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.210987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.210987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.211059 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.211668 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.211669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.211730 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.211827 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.211963 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:29:40.214288 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: Bootstrap starting.
I20260925 15:29:40.215097 24024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:40.215629 24024 log.cc:824] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:40.217128 24024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: No bootstrap required, opened a new log
Sep 25 15:29:40 dist-test-slave-hvv5 krb5kdc[23868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:40.219765 24024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.219931 24024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a8a64415224c81a111814b3328408e, State: Initialized, Role: FOLLOWER
I20260925 15:29:40.220023 24024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.220419 24120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.220485 24024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:40.220520 24120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.222168 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.211614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:40779 (local address 127.0.0.1:45800)
0925 15:29:40.211927 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:40.211941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:40.211984 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:40.212361 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:40.212362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:40.212569 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:40.212631 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.212635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.214102 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.214104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.214881 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:40.214885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.215961 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.215962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:40.216036 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.216607 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:40.216613 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:40.218168 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:40.220804 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.220806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.220806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.221060 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.221291 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:40.221291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:40.221292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:40.221348 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:40.221755 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:40.221756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:40.221841 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 15:29:40.221915 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:29:40.222047 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
W20260925 15:29:40.222528 24131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:40.222584 24131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:40.222965 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.211690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.1.163.124:40779)
0925 15:29:40.212144 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.212146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.212154 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:40.212204 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.212205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.212219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.212721 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.212795 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.213954 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.215034 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.215794 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.216212 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.218274 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.218276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.218277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.218286 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.220692 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.221155 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.221156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.221157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.221221 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.221430 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.221431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.221431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.221618 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.221702 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.222729 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.222840 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":144,"threads_started":1}
I20260925 15:29:40.224979 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: Bootstrap starting.
I20260925 15:29:40.226068 23949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:40.222623 23956 master_runner.cc:186] Error getting master registration for 127.1.163.126:38337: OK, instance_id { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" instance_seqno: 1790350179757900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:29:40.226194 23956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:29:40.226428 23949 log.cc:824] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:40.227650 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: No bootstrap required, opened a new log
I20260925 15:29:40.229945 23949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.230124 23949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325d6f983e864fc4b9266fc3226bf317, State: Initialized, Role: FOLLOWER
I20260925 15:29:40.230222 23949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.230573 23949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:40.230600 23923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:29:40.230675 23923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:40.231618 23923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
I20260925 15:29:40.231918 23923 leader_election.cc:288] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [CANDIDATE]: Term 1 election: Requested vote from peers 37a8a64415224c81a111814b3328408e (127.1.163.125:38715), df53d04f3bf34dd9ab6d9d165bcb56cf (127.1.163.124:40779)
I20260925 15:29:40.234330 23998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "325d6f983e864fc4b9266fc3226bf317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "37a8a64415224c81a111814b3328408e"
I20260925 15:29:40.234510 23998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:40.235567 23998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 325d6f983e864fc4b9266fc3226bf317 in term 1.
I20260925 15:29:40.236081 24073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "325d6f983e864fc4b9266fc3226bf317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf"
I20260925 15:29:40.236359 24073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:40.237037 23900 leader_election.cc:302] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 325d6f983e864fc4b9266fc3226bf317, 37a8a64415224c81a111814b3328408e; no voters:
I20260925 15:29:40.238437 24073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 325d6f983e864fc4b9266fc3226bf317 in term 1.
I20260925 15:29:40.239953 24134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.240836 23881 master_runner.cc:263] No leader master found from master 325d6f983e864fc4b9266fc3226bf317
I20260925 15:29:40.240904 23881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:29:40.240826 24134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.241099 24134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:40.241206 24134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [term 1 LEADER]: Becoming Leader. State: Replica: 325d6f983e864fc4b9266fc3226bf317, State: Running, Role: LEADER
I20260925 15:29:40.241357 24134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } }
W20260925 15:29:40.241436 24145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:40.241830 24145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:40.246742 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.234054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.1.163.126:38337)
0925 15:29:40.234100 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.234102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.237199 (+ 3097us) server_negotiation.cc:407] Connection header received
0925 15:29:40.237324 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.237325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.237349 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.237385 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.238293 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.239145 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.240485 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.240825 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.240877 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.242690 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.242691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.242692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.242700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.244919 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.245779 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.245781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.245782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.245849 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.246075 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.246075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.246076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.246212 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.246286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.246537 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.246624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:29:40.252833 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325d6f983e864fc4b9266fc3226bf317. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.252997 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:40.253286 24156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:40.253504 24156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 15:29:40.254937 23902 tablet.cc:2369] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:40.256004 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.240386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.1.163.125:38715)
0925 15:29:40.240474 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.240476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.244729 (+ 4253us) server_negotiation.cc:407] Connection header received
0925 15:29:40.244844 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.244846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.244857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.244889 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.246943 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.247805 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.249740 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.250151 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.250214 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.252095 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.252097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.252097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.252107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.254023 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.254604 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.254606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.254606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.254810 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.255097 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.255098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.255099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.255309 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.255412 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.255789 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.255902 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:29:40.261474 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:40.240755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.1.163.124:40779)
0925 15:29:40.240795 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.240797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.246097 (+ 5300us) server_negotiation.cc:407] Connection header received
0925 15:29:40.247373 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.247375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.247394 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.247439 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.253209 (+ 5770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.253933 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.255619 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:40.255981 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:40.256027 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:40.258273 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:40.258275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:40.258275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:40.258283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:40.260061 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.260487 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.260488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.260488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.260593 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:40.260803 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:40.260803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:40.260804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:40.260956 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:40.261070 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:40.261283 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:29:40.261376 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:29:40.261946 24073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [term 1 FOLLOWER]: Refusing update from remote peer 325d6f983e864fc4b9266fc3226bf317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:40.262363 24151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:40.262539 23998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [term 1 FOLLOWER]: Refusing update from remote peer 325d6f983e864fc4b9266fc3226bf317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:40.264035 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325d6f983e864fc4b9266fc3226bf317. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.264118 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.264182 24151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:40.265035 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.265129 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:40.265527 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.265617 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.265275 24157 mvcc.cc:205] Tried to move back new op lower bound from 7333274338349219840 to 7333274338271014912. Current Snapshot: MvccSnapshot[applied={T|T < 7333274338349219840}]
I20260925 15:29:40.266157 24120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325d6f983e864fc4b9266fc3226bf317. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.266253 24120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.266784 24134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.266892 24134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317 [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:40.267640 24120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "325d6f983e864fc4b9266fc3226bf317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "325d6f983e864fc4b9266fc3226bf317" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 38337 } } peers { permanent_uuid: "37a8a64415224c81a111814b3328408e" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 38715 } } peers { permanent_uuid: "df53d04f3bf34dd9ab6d9d165bcb56cf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 40779 } } }
I20260925 15:29:40.267728 24120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:40.269086 24156 catalog_manager.cc:1347] Generated new cluster ID: 4a53aee298da48b7b223dd044125e420
I20260925 15:29:40.269148 24156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:40.269539 24158 mvcc.cc:205] Tried to move back new op lower bound from 7333274338349219840 to 7333274338271014912. Current Snapshot: MvccSnapshot[applied={T|T < 7333274338349219840}]
I20260925 15:29:40.273435 24161 mvcc.cc:205] Tried to move back new op lower bound from 7333274338349219840 to 7333274338271014912. Current Snapshot: MvccSnapshot[applied={T|T < 7333274338349219840}]
I20260925 15:29:40.303602 24156 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:40.304440 24156 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:40.314793 24156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325d6f983e864fc4b9266fc3226bf317: Generated new TSK 0
I20260925 15:29:40.315138 24156 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:29:40.349825 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.347780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.1.163.125:38715)
0925 15:29:40.347833 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.347836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.349059 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 15:29:40.349184 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.349185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.349197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.349230 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.349733 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49780: BlockingRecv error: recv got EOF from 127.0.0.1:49780 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 15:29:40.351433 24118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.348173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.1.163.124:40779)
0925 15:29:40.348214 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.348216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.350149 (+ 1933us) server_negotiation.cc:407] Connection header received
0925 15:29:40.350709 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.350711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.350725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.350758 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.351360 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45824: BlockingRecv error: recv got EOF from 127.0.0.1:45824 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:29:40.352269 23952 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.348633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.1.163.126:38337)
0925 15:29:40.348678 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.348680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.350644 (+ 1964us) server_negotiation.cc:407] Connection header received
0925 15:29:40.351758 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.351759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.351772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.351805 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.352202 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52004: BlockingRecv error: recv got EOF from 127.0.0.1:52004 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:29:40.354861 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.354027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.1.163.125:38715)
0925 15:29:40.354073 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.354074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.354291 (+ 217us) server_negotiation.cc:407] Connection header received
0925 15:29:40.354403 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.354405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.354425 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.354455 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.354792 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49794: BlockingRecv error: recv got EOF from 127.0.0.1:49794 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:29:40.358819 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.357532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.1.163.125:38715)
0925 15:29:40.357595 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.357597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.357967 (+ 370us) server_negotiation.cc:407] Connection header received
0925 15:29:40.358339 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.358341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.358353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.358384 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.358763 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49804: BlockingRecv error: recv got EOF from 127.0.0.1:49804 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:29:40.362900 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.362021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.1.163.125:38715)
0925 15:29:40.362064 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.362066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.362321 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:29:40.362419 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.362420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.362432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.362467 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.362840 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49810: BlockingRecv error: recv got EOF from 127.0.0.1:49810 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:29:40.369431 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.368411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.1.163.125:38715)
0925 15:29:40.368459 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.368461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.368758 (+ 297us) server_negotiation.cc:407] Connection header received
0925 15:29:40.368892 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.368894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.368908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.368942 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.369367 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49820: BlockingRecv error: recv got EOF from 127.0.0.1:49820 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 15:29:40.380446 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.378991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.1.163.125:38715)
0925 15:29:40.379046 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.379048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.379416 (+ 368us) server_negotiation.cc:407] Connection header received
0925 15:29:40.379543 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.379545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.379559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.379594 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.380379 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49830: BlockingRecv error: recv got EOF from 127.0.0.1:49830 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 15:29:40.399098 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.398053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.1.163.125:38715)
0925 15:29:40.398105 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.398107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.398450 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:29:40.398561 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.398562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.398574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.398606 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.399033 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49844: BlockingRecv error: recv got EOF from 127.0.0.1:49844 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 15:29:40.433418 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.432561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.1.163.125:38715)
0925 15:29:40.432614 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.432616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.432787 (+ 171us) server_negotiation.cc:407] Connection header received
0925 15:29:40.432850 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.432852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.432868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.432937 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.433352 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49854: BlockingRecv error: recv got EOF from 127.0.0.1:49854 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 15:29:40.499511 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.498687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.1.163.125:38715)
0925 15:29:40.498826 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.498829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.498856 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:29:40.498944 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.498945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.498957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.499025 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.499444 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49862: BlockingRecv error: recv got EOF from 127.0.0.1:49862 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 15:29:40.629945 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.628998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.1.163.125:38715)
0925 15:29:40.629089 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.629092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.629166 (+ 74us) server_negotiation.cc:407] Connection header received
0925 15:29:40.629267 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.629269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.629282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.629368 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.629854 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49870: BlockingRecv error: recv got EOF from 127.0.0.1:49870 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:29:40.888252 24028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:40.887532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.1.163.125:38715)
0925 15:29:40.887610 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:40.887613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:40.887623 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:40.887659 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:40.887660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:40.887669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:40.887738 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:40.888168 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49876: BlockingRecv error: recv got EOF from 127.0.0.1:49876 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:29:41.192891 24116 catalog_manager.cc:1259] Loaded cluster ID: 4a53aee298da48b7b223dd044125e420
I20260925 15:29:41.192977 24116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: loading cluster ID for follower catalog manager: success
I20260925 15:29:41.193792 24116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: acquiring CA information for follower catalog manager: success
I20260925 15:29:41.194073 24116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df53d04f3bf34dd9ab6d9d165bcb56cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:29:41.223090 24131 catalog_manager.cc:1259] Loaded cluster ID: 4a53aee298da48b7b223dd044125e420
I20260925 15:29:41.223392 24131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: loading cluster ID for follower catalog manager: success
I20260925 15:29:41.224360 24131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: acquiring CA information for follower catalog manager: success
I20260925 15:29:41.224802 24131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 37a8a64415224c81a111814b3328408e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:29:41.404778 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:41.401773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.1.163.125:38715)
0925 15:29:41.401956 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:41.401958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:41.401969 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:29:41.401998 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:41.401999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:41.402052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:41.402137 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:41.402371 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.402892 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.403792 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.404009 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.404081 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:41.404286 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:41.404407 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:41.404484 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:41.404618 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 15:29:41.404642 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260925 15:29:41.409466 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:41.407007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.1.163.124:40779)
0925 15:29:41.407163 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:41.407165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:41.407174 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:41.407221 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:41.407222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:41.407261 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:41.407346 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:41.407453 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.407944 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.408733 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.408974 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.409039 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:41.409166 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:41.409276 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:41.409344 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:41.409376 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:29:41.409396 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260925 15:29:41.414507 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:41.411523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.1.163.124:40779)
0925 15:29:41.411566 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:41.411568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:41.411696 (+ 128us) server_negotiation.cc:407] Connection header received
0925 15:29:41.411722 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:41.411723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:41.411744 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:41.411789 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:41.411993 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.412759 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.413574 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.413888 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.413949 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:41.414103 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:41.414139 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:41.414186 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:41.414406 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:29:41.414432 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:29:41.414541 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:41.411244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.1.163.126:38337)
0925 15:29:41.411358 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:41.411360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:41.411484 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:29:41.411533 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:41.411535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:41.411587 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:41.411662 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:41.411880 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.412427 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.413479 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.413805 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.413871 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:41.414056 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:41.414173 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:41.414241 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:41.414298 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:29:41.414326 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260925 15:29:41.414625 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:41.411284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.1.163.125:38715)
0925 15:29:41.411329 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:41.411331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:41.411470 (+ 139us) server_negotiation.cc:407] Connection header received
0925 15:29:41.411561 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:41.411562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:41.411574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:41.411627 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:41.411775 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.412714 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.413757 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:41.414040 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:41.414108 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:41.414286 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:41.414321 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:41.414366 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:41.414521 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:29:41.414544 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:29:41.416390 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23881
I20260925 15:29:41.422143 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 23956
I20260925 15:29:41.428177 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 24031
2026-09-25T15:29:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3087 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:29:41 dist-test-slave-hvv5 krb5kdc[24247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:29:41 dist-test-slave-hvv5 krb5kdc[24247](info): set up 2 sockets
Sep 25 15:29:41 dist-test-slave-hvv5 krb5kdc[24247](info): commencing operation
krb5kdc: starting...
W20260925 15:29:42.541039 1677 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/test-user.keytab.
Sep 25 15:29:42 dist-test-slave-hvv5 krb5kdc[24247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:29:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.163.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
Entry for principal kudu/127.1.163.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab.
I20260925 15:29:42.677891 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37549
--webserver_interface=127.1.163.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:29:42.764166 24260 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:42.764509 24260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:42.764544 24260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:42.764616 24260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:42.766305 24260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:42.766429 24260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:42.766460 24260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:42.766481 24260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:42.766510 24260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:42.768491 24260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.126.keytab
--principal=kudu/127.1.163.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.126:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.163.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:42.768882 24260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:42.769150 24260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:42.774616 24268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:42.774616 24266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:42.774735 24265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:42.775478 24260 server_base.cc:1034] running on GCE node
I20260925 15:29:42.775774 24260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:42.776130 24260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:42.777491 24260 hybrid_clock.cc:648] HybridClock initialized: now 1790350182777474 us; error 33 us; skew 500 ppm
Sep 25 15:29:42 dist-test-slave-hvv5 krb5kdc[24247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:42.787597 24260 init.cc:377] Logged in from keytab as kudu/127.1.163.126@KRBTEST.COM (short username kudu)
I20260925 15:29:42.788625 24260 webserver.cc:460] Webserver started at http://127.1.163.126:38967/ using document root <none> and password file <none>
I20260925 15:29:42.788890 24260 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:42.788968 24260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:42.789131 24260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:42.790556 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data/instance:
uuid: "5932fe6911c24f3fab34cf33c2c40b9b"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "4ac9704c032c97ea0dfb1fe1d270baab"
server_key_iv: "a472de1a42b88d1a4d82773ee738ad46"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.791134 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance:
uuid: "5932fe6911c24f3fab34cf33c2c40b9b"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "4ac9704c032c97ea0dfb1fe1d270baab"
server_key_iv: "a472de1a42b88d1a4d82773ee738ad46"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.792733 24260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:29:42.793761 24275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:42.794099 24260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:42.794245 24260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal
uuid: "5932fe6911c24f3fab34cf33c2c40b9b"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "4ac9704c032c97ea0dfb1fe1d270baab"
server_key_iv: "a472de1a42b88d1a4d82773ee738ad46"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.794349 24260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:42.806433 24260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:42.809420 24260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:42.809648 24260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:42.815519 24260 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.126:37549
I20260925 15:29:42.815536 24327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.126:37549 every 8 connection(s)
I20260925 15:29:42.816102 24260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/data/info.pb
I20260925 15:29:42.817550 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:42.819433 24328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:42.824069 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 24260
I20260925 15:29:42.824182 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-0/wal/instance
I20260925 15:29:42.824378 1677 external_mini_cluster.cc:1428] Setting key 60e35a662906bdc027d135cbf85a9081
Sep 25 15:29:42 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.125@KRBTEST.COM; defaulting to no policy
I20260925 15:29:42.832291 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.820354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.1.163.126:37549)
0925 15:29:42.820575 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:42.820627 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:42.820664 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:29:42.821239 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:42.821242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:42.821263 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:42.821361 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:42.821954 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:42.822924 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:42.824072 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:42.824670 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:42.824788 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:42.828281 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:42.828293 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:42.828293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:42.828301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:42.830297 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:42.830798 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:42.830802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:42.830802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:42.830898 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:42.831155 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:42.831156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:42.831157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:42.831506 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:42.831627 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:42.831842 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:29:42.832003 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260925 15:29:42.832299 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.820299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37549 (local address 127.0.0.1:39326)
0925 15:29:42.820518 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.820588 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:42.821076 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:42.821367 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:42.821369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:42.821640 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:42.821810 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:42.821814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:42.823050 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:42.823052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:42.823958 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:42.823961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:42.824773 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:42.824774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:42.824956 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:42.825584 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:42.825593 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:42.828101 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:42.830434 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:42.830441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:42.830445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:42.830682 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:42.830977 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:42.830978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:42.830978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:42.831080 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:42.831599 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:42.831605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:42.831678 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 15:29:42.831802 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:29:42.832075 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":101,"thread_start_us":128,"threads_started":1}
I20260925 15:29:42.833326 24328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:42.833796 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.833618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55606)
0925 15:29:42.833717 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.833770 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 15:29:42.834043 24278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
I20260925 15:29:42.834713 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.834602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55612)
0925 15:29:42.834661 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.834691 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:29:42.834841 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:43197: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
Entry for principal kudu/127.1.163.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab.
I20260925 15:29:42.865414 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:43197
--webserver_interface=127.1.163.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:29:42.884124 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } attempt: 1
I20260925 15:29:42.884861 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.884683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55614)
0925 15:29:42.884777 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.884827 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:29:42.885345 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.885184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55630)
0925 15:29:42.885267 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.885308 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 15:29:42.885488 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:43197: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
I20260925 15:29:42.953696 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } attempt: 2
I20260925 15:29:42.954222 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.954085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55640)
0925 15:29:42.954154 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.954193 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:29:42.954698 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:42.954565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55646)
0925 15:29:42.954642 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:42.954676 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:29:42.954818 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.125:43197: Network error: Client connection negotiation failed: client connection to 127.1.163.125:43197: connect: Connection refused (error 111)
W20260925 15:29:42.969439 24335 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:42.969854 24335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:42.969944 24335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:42.969986 24335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:42.971467 24335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:42.971513 24335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:42.971529 24335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:42.971542 24335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:42.971554 24335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:42.973418 24335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.125.keytab
--principal=kudu/127.1.163.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.125:43197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.163.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:42.973827 24335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:42.974149 24335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:42.980521 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:42.980516 24341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:42.980744 24340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:42.981263 24335 server_base.cc:1034] running on GCE node
I20260925 15:29:42.981511 24335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:42.981854 24335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:42.983335 24335 hybrid_clock.cc:648] HybridClock initialized: now 1790350182983220 us; error 81 us; skew 500 ppm
Sep 25 15:29:42 dist-test-slave-hvv5 krb5kdc[24247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:42.987108 24335 init.cc:377] Logged in from keytab as kudu/127.1.163.125@KRBTEST.COM (short username kudu)
I20260925 15:29:42.987697 24335 webserver.cc:460] Webserver started at http://127.1.163.125:35607/ using document root <none> and password file <none>
I20260925 15:29:42.987970 24335 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:42.988042 24335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:42.988204 24335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:42.989778 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data/instance:
uuid: "4d73ea1afb464dbc9b2695b852796dcf"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "6a90dd351f256415617a8a1a69da72b9"
server_key_iv: "29d315504f06b646d983558e78eab410"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.990403 24335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal/instance:
uuid: "4d73ea1afb464dbc9b2695b852796dcf"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "6a90dd351f256415617a8a1a69da72b9"
server_key_iv: "29d315504f06b646d983558e78eab410"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.992230 24335 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:29:42.993520 24350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:42.993806 24335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:29:42.993922 24335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal
uuid: "4d73ea1afb464dbc9b2695b852796dcf"
format_stamp: "Formatted at 2026-09-25 15:29:42 on dist-test-slave-hvv5"
server_key: "6a90dd351f256415617a8a1a69da72b9"
server_key_iv: "29d315504f06b646d983558e78eab410"
server_key_version: "encryptionkey@0"
I20260925 15:29:42.994144 24335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:43.021504 24335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:43.024268 24335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:43.024468 24335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:43.029659 24335 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.125:43197
I20260925 15:29:43.029714 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.125:43197 every 8 connection(s)
I20260925 15:29:43.030143 24335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/data/info.pb
I20260925 15:29:43.031364 24403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:43.032801 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 24335
I20260925 15:29:43.032935 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-1/wal/instance
I20260925 15:29:43.033277 1677 external_mini_cluster.cc:1428] Setting key 40baf71f350f4e3f4b50a03043f05893
I20260925 15:29:43.033943 24403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.163.124@KRBTEST.COM; defaulting to no policy
I20260925 15:29:43.045279 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.034770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.1.163.126:37549)
0925 15:29:43.034842 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.034844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.035123 (+ 279us) server_negotiation.cc:407] Connection header received
0925 15:29:43.035678 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.035679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.035697 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.035749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.036462 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.037143 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.038323 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.038559 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.038650 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.041486 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.041488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.041488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.041498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.043598 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.044156 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.044158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.044158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.044228 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.044515 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.044516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.044516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.044692 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.044772 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.045073 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.045178 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:29:43.045675 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.034777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37549 (local address 127.0.0.1:39336)
0925 15:29:43.034966 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.035010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.035555 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.035842 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.035845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.036112 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.036336 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.036341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.037276 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.037278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.038228 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.038230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.038749 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.038753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.038902 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.039638 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.039648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.041360 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.043752 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.043764 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.043767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.044044 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.044302 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.044303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.044304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.044431 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.044763 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.044771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.044931 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.045053 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.045243 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260925 15:29:43.045974 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } attempt: 3
I20260925 15:29:43.046886 24403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.163.124@KRBTEST.COM" created.
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:29:43.056355 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.046362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.1.163.125:43197)
0925 15:29:43.046810 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.046815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.046841 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:29:43.046867 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.046870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.046887 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.046978 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.047445 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.048403 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.049472 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.049959 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.050348 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.052343 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.052355 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.052355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.052364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.054668 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.055138 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.055143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.055143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.055216 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.055461 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.055461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.055462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.055838 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.055940 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.056174 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.056258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":122,"threads_started":1}
I20260925 15:29:43.056372 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.046293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55650)
0925 15:29:43.046347 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.046359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.046413 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.047126 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.047127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.047286 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.047357 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.047359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.048500 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.048501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.049387 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.049390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.050056 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.050057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.050145 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.050663 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.050669 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.052249 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.054779 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.054781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.054781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.055036 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.055300 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.055301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.055301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.055384 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.055931 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.055933 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.055998 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.056083 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.056215 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:29:43.057416 24328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:43.057786 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.047706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55660)
0925 15:29:43.047822 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.047837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.047891 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.048099 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.048100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.048405 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.048480 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.048483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.049458 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.049459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.050162 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.050165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.050646 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.050648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.050714 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.051106 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.051114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.052930 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.055438 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.055439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.055439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.055689 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.056622 (+ 933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.056623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.056623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.056691 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.057454 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.057455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.057499 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.057557 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.057661 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:29:43.058116 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.057971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56228)
0925 15:29:43.058046 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.058083 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:29:43.058461 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.047769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.1.163.125:43197)
0925 15:29:43.047948 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.047950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.047958 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:43.047978 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.047979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.047990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.048073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.048608 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.049362 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.050311 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.050531 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.050643 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.053019 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.053021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.053021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.053028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.055355 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.055783 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.055784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.055785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.055843 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.057131 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.057132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.057132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.057289 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.057420 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.057968 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.058071 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260925 15:29:43.058833 24403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:43.059295 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.059105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56244)
0925 15:29:43.059209 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.059268 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:29:43.059401 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.059012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56242)
0925 15:29:43.059340 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.059373 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
W20260925 15:29:43.059460 24352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
W20260925 15:29:43.059567 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
I20260925 15:29:43.060122 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.059995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56254)
0925 15:29:43.060068 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.060099 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:29:43.060295 24403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
Entry for principal kudu/127.1.163.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab.
I20260925 15:29:43.081166 1677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk86Bl6/build/release/bin/kudu
/tmp/dist-test-taskk86Bl6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:35309
--webserver_interface=127.1.163.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.163.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:29:43.090806 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 1
I20260925 15:29:43.091485 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.091342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56258)
0925 15:29:43.091408 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.091452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:29:43.091885 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.091785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56274)
0925 15:29:43.091832 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.091860 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:29:43.091979 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
I20260925 15:29:43.109660 24403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 1
I20260925 15:29:43.110368 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.110184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56288)
0925 15:29:43.110254 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.110336 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:29:43.110889 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.110739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56292)
0925 15:29:43.110832 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.110865 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 15:29:43.111152 24403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
I20260925 15:29:43.167362 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 2
I20260925 15:29:43.167944 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.167726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56296)
0925 15:29:43.167820 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.167894 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:29:43.168711 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.168377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56310)
0925 15:29:43.168443 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.168626 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:29:43.168972 24328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
I20260925 15:29:43.179301 24403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 2
I20260925 15:29:43.179962 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.179760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56312)
0925 15:29:43.179842 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.179911 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:29:43.180364 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.180251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56318)
0925 15:29:43.180308 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.180337 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:29:43.180616 24403 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.163.124:35309: Network error: Client connection negotiation failed: client connection to 127.1.163.124:35309: connect: Connection refused (error 111)
W20260925 15:29:43.182808 24411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:29:43.183161 24411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:29:43.183209 24411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:29:43.183266 24411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:29:43.185134 24411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:29:43.185190 24411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:29:43.185206 24411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:29:43.185220 24411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:29:43.185233 24411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:29:43.187939 24411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.163.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.163.126:37549,127.1.163.125:43197,127.1.163.124:35309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/krb5kdc/kudu_127.1.163.124.keytab
--principal=kudu/127.1.163.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.163.124:35309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.163.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:29:43.188272 24411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:29:43.188494 24411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:29:43.194284 24419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:43.194578 24416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:29:43.194589 24417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:29:43.194787 24411 server_base.cc:1034] running on GCE node
I20260925 15:29:43.195108 24411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:29:43.195518 24411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:29:43.196769 24411 hybrid_clock.cc:648] HybridClock initialized: now 1790350183196697 us; error 114 us; skew 500 ppm
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:29:43.200256 24411 init.cc:377] Logged in from keytab as kudu/127.1.163.124@KRBTEST.COM (short username kudu)
I20260925 15:29:43.200889 24411 webserver.cc:460] Webserver started at http://127.1.163.124:40421/ using document root <none> and password file <none>
I20260925 15:29:43.201149 24411 fs_manager.cc:359] Metadata directory not provided
I20260925 15:29:43.201224 24411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:29:43.201376 24411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:29:43.202498 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data/instance:
uuid: "09ff55e4d00946d08401292b241fe0bf"
format_stamp: "Formatted at 2026-09-25 15:29:43 on dist-test-slave-hvv5"
server_key: "1396a0666237c30356049de280417229"
server_key_iv: "b969a68ec7d41d199ec37e1d404b22b8"
server_key_version: "encryptionkey@0"
I20260925 15:29:43.202890 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal/instance:
uuid: "09ff55e4d00946d08401292b241fe0bf"
format_stamp: "Formatted at 2026-09-25 15:29:43 on dist-test-slave-hvv5"
server_key: "1396a0666237c30356049de280417229"
server_key_iv: "b969a68ec7d41d199ec37e1d404b22b8"
server_key_version: "encryptionkey@0"
I20260925 15:29:43.204517 24411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:29:43.205677 24426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:29:43.205942 24411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:29:43.206125 24411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data,/tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal
uuid: "09ff55e4d00946d08401292b241fe0bf"
format_stamp: "Formatted at 2026-09-25 15:29:43 on dist-test-slave-hvv5"
server_key: "1396a0666237c30356049de280417229"
server_key_iv: "b969a68ec7d41d199ec37e1d404b22b8"
server_key_version: "encryptionkey@0"
I20260925 15:29:43.206200 24411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:29:43.241827 24411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:29:43.244848 24411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:29:43.245116 24411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:29:43.250532 24411 rpc_server.cc:307] RPC server started. Bound to: 127.1.163.124:35309
I20260925 15:29:43.250723 24478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.163.124:35309 every 8 connection(s)
I20260925 15:29:43.251036 24411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/data/info.pb
I20260925 15:29:43.252236 24479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:29:43.254037 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:43.258260 1677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk86Bl6/build/release/bin/kudu as pid 24411
I20260925 15:29:43.258456 1677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk86Bl6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349904034090-1677-0/minicluster-data/master-2/wal/instance
I20260925 15:29:43.258682 1677 external_mini_cluster.cc:1428] Setting key 39bc8a4c481de9297c2eb7c8aa6b5803
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:43.266358 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.254923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.1.163.126:37549)
0925 15:29:43.255035 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.255038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.255208 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:29:43.255842 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.255844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.255856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.255900 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.256656 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.257157 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.258986 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.259184 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.259302 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.262422 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.262424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.262424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.262431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.264673 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.265147 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.265150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.265150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.265218 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.265510 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.265511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.265511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.265639 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.265840 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.266153 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.266248 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:29:43.266580 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.254803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.126:37549 (local address 127.0.0.1:39350)
0925 15:29:43.255077 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.255117 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.255697 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.255984 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.255988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.256264 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.256485 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.256490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.257396 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.257397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.258868 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.258873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.259562 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.259564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.259724 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.260392 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.260402 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.262302 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.264790 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.264800 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.264805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.265062 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.265309 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.265310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.265310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.265438 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.265831 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.265840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.266001 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.266125 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.266306 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":127,"threads_started":1}
I20260925 15:29:43.267601 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:43.268180 24328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 3
I20260925 15:29:43.270745 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.259335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.1.163.124:35309)
0925 15:29:43.259550 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.259557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.259578 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:29:43.259599 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.259601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.259623 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.259709 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.260575 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.261483 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.262488 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.262904 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.262971 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.266330 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.266337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.266338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.266345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.269267 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.269731 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.269734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.269735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.269814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.270042 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.270043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.270043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.270253 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.270447 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.270511 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.270625 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":143,"threads_started":1}
I20260925 15:29:43.271812 24403 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } attempt: 3
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.126@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.125@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:43.279469 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.268043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.1.163.125:43197)
0925 15:29:43.268113 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.268115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.268370 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:29:43.268490 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.268492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.268508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.268633 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.269085 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.270094 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271035 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.271333 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.275629 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.275631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.275631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.275639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.277630 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278431 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.278432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.278433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.278503 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278833 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.278834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.278835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.278981 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.279028 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.279253 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.279356 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:29:43.279711 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.268570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56342)
0925 15:29:43.268622 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.268633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.268686 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.269165 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.269166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.269333 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.269384 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.269386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.270256 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.270257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.271135 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.271139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.271617 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.271689 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.272663 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.272669 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.275073 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.277980 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.277981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.277982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.278256 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.278483 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.278484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.278552 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.279364 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.279366 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.279421 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.279478 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.279602 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:29:43.280200 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.267964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.125:43197 (local address 127.0.0.1:55674)
0925 15:29:43.268308 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.268320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.268399 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.268714 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.268716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.268938 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.268985 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.268987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.270179 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.270180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.270958 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.270961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.271409 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.271492 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.272598 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.272604 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.275534 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.277797 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.277799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.277800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.278216 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.278597 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.278598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.278676 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.279091 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.279092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.279147 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.279217 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.280080 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":3}
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.126@KRBTEST.COM
I20260925 15:29:43.280368 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.268614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.1.163.124:35309)
0925 15:29:43.268979 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.268981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.268989 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:43.269010 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.269011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.269022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.269132 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.269471 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.270175 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271212 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.271536 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.271604 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.275189 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.275191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.275191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.275201 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.277867 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278344 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.278344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.278345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.278402 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.278895 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.278896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.278896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.279064 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.279834 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.279876 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.280181 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":341,"thread_start_us":93,"threads_started":1}
I20260925 15:29:43.280961 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:29:43.282486 24328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: Bootstrap starting.
I20260925 15:29:43.283169 24328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:43.283687 24328 log.cc:824] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:43.284391 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.272481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.1.163.126:37549)
0925 15:29:43.272525 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.272526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.272717 (+ 191us) server_negotiation.cc:407] Connection header received
0925 15:29:43.272893 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.272896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.272917 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.272966 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.273894 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.274441 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.275245 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.276515 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.276592 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.281028 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.281030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.281030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.281037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.282730 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.283167 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.283168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.283168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.283293 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.283514 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.283515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.283515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.283753 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.283913 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.284160 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.284264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:29:43.284739 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.272146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.1.163.124:35309)
0925 15:29:43.272318 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.272321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.272329 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:43.272350 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.272350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.272363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.272414 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.272832 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.273611 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.274773 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.275147 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.275210 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.278054 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.278055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.278055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.278063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.279776 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.280175 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.280176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.280176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.280230 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.280637 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.280638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.280639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.280782 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.280842 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.284445 (+ 3603us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.284543 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260925 15:29:43.284875 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.272124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56352)
0925 15:29:43.272180 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.272191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.272231 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.272460 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.272461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.272684 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.272735 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.272737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.273721 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.273722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.274652 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.274656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.275283 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.275284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.275349 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.276031 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.276037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.277962 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.279870 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.279871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.279871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.280060 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.280444 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.280445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.280446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.280507 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.284007 (+ 3500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.284008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.284063 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.284127 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.284244 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:29:43.284982 24328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: No bootstrap required, opened a new log
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.163.124@KRBTEST.COM for kudu/127.1.163.124@KRBTEST.COM
I20260925 15:29:43.286880 24403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: Bootstrap starting.
I20260925 15:29:43.287611 24328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.287904 24328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5932fe6911c24f3fab34cf33c2c40b9b, State: Initialized, Role: FOLLOWER
I20260925 15:29:43.288049 24403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:43.288015 24328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.288383 24328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:43.288740 24403 log.cc:824] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:43.289990 24403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: No bootstrap required, opened a new log
I20260925 15:29:43.291213 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.281325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.1.163.124:35309)
0925 15:29:43.281386 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.281388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.281396 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:29:43.281417 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.281418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.281428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.281491 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.281802 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.282513 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.283543 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.283750 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.283830 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.287105 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.287106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.287107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.287114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.289749 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.290153 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.290154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.290155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.290392 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.290651 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.290652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.290652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.290780 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.290827 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.291022 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.291121 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:29:43.291538 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.281258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.163.124:35309 (local address 127.0.0.1:56360)
0925 15:29:43.281301 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:29:43.281312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:29:43.281349 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:29:43.281493 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:29:43.281493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:29:43.281663 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:29:43.281708 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.281710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.282654 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.282655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.283467 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:29:43.283469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.283878 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.283878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:29:43.283927 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.284651 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:29:43.284656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:29:43.286935 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:29:43.289844 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.289845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.289845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.290073 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.290498 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:29:43.290499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:29:43.290500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:29:43.290578 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:29:43.290871 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:29:43.290872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:29:43.290917 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:29:43.291349 (+ 432us) client_negotiation.cc:240] Negotiation successful
0925 15:29:43.291455 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
Sep 25 15:29:43 dist-test-slave-hvv5 krb5kdc[24247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790350182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.163.125@KRBTEST.COM
I20260925 15:29:43.291946 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.292408 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.292392 24403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.293180 24403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d73ea1afb464dbc9b2695b852796dcf, State: Initialized, Role: FOLLOWER
I20260925 15:29:43.293344 24403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.293717 24403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:43.294831 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: Bootstrap starting.
I20260925 15:29:43.291568 24260 master_runner.cc:186] Error getting master registration for 127.1.163.125:43197: OK, instance_id { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" instance_seqno: 1790350183028172 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:29:43.295393 24260 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:29:43.295640 24479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: Neither blocks nor log segments found. Creating new log.
I20260925 15:29:43.296450 24479 log.cc:824] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:29:43.297497 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.285521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.1.163.125:43197)
0925 15:29:43.285557 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.285559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.285633 (+ 74us) server_negotiation.cc:407] Connection header received
0925 15:29:43.285740 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.285741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.285751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.285780 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.286692 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.287394 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.288609 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.288890 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.288967 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.292949 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.292951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.292951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.292961 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.295239 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.295937 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.295939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.295940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.296021 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.296334 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.296335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.296336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.296490 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.296721 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.297250 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.297359 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:29:43.297585 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: No bootstrap required, opened a new log
W20260925 15:29:43.297384 24510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:43.297804 24510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:43.297951 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.298032 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.299357 1677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:29:43.300213 24479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.300450 24479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff55e4d00946d08401292b241fe0bf, State: Initialized, Role: FOLLOWER
I20260925 15:29:43.300464 24302 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5932fe6911c24f3fab34cf33c2c40b9b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39364
I20260925 15:29:43.300518 24479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.300647 24302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:29:43.300709 24302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:43.300899 24479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:29:43.301781 24302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.302033 24302 leader_election.cc:288] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [CANDIDATE]: Term 1 election: Requested vote from peers 4d73ea1afb464dbc9b2695b852796dcf (127.1.163.125:43197), 09ff55e4d00946d08401292b241fe0bf (127.1.163.124:35309)
I20260925 15:29:43.302649 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09ff55e4d00946d08401292b241fe0bf"
I20260925 15:29:43.302765 24453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:43.303700 24453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5932fe6911c24f3fab34cf33c2c40b9b in term 1.
I20260925 15:29:43.304157 24277 leader_election.cc:302] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff55e4d00946d08401292b241fe0bf, 5932fe6911c24f3fab34cf33c2c40b9b; no voters:
I20260925 15:29:43.304174 24377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d73ea1afb464dbc9b2695b852796dcf"
I20260925 15:29:43.304291 24377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:29:43.304322 24486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:29:43.304454 24486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [term 1 LEADER]: Becoming Leader. State: Replica: 5932fe6911c24f3fab34cf33c2c40b9b, State: Running, Role: LEADER
I20260925 15:29:43.304610 24486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } }
I20260925 15:29:43.305223 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5932fe6911c24f3fab34cf33c2c40b9b. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.305370 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: This master's current role is: LEADER
W20260925 15:29:43.305676 24498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:43.305858 24498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:29:43.306046 24377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5932fe6911c24f3fab34cf33c2c40b9b in term 1.
I20260925 15:29:43.301622 24335 master_runner.cc:186] Error getting master registration for 127.1.163.124:35309: OK, instance_id { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" instance_seqno: 1790350183248795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:29:43.306301 24335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:29:43.306936 24513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.307096 24513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:29:43.308113 24524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:29:43.308172 24524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 15:29:43.310549 24281 tablet.cc:2369] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:29:43.320777 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.310398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.1.163.126:37549)
0925 15:29:43.310473 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.310475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.310839 (+ 364us) server_negotiation.cc:407] Connection header received
0925 15:29:43.310979 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.310980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.310993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.311034 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.311959 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.312853 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.314050 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.314385 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.314435 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.316489 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.316491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.316491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.316500 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.319142 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.319645 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.319646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.319647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.319711 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.319910 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.319910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.319911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.320072 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.320248 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.320579 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.320676 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:29:43.327909 24532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:29:43.328214 24532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:29:43.330350 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.309928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.1.163.125:43197)
0925 15:29:43.309972 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.309974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.320737 (+ 10763us) server_negotiation.cc:407] Connection header received
0925 15:29:43.321082 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.321084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.321098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.321138 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.321958 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.322765 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.323939 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.324380 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.324436 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.326805 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.326807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.326807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.326817 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.328716 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.329293 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.329295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.329295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.329360 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.329606 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.329607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.329607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.329763 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.329855 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.330145 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.330240 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:29:43.332901 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:43.310152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.1.163.124:35309)
0925 15:29:43.310191 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.310194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.320736 (+ 10542us) server_negotiation.cc:407] Connection header received
0925 15:29:43.320883 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.320885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.320899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.320934 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.322581 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.324086 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.325240 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:43.325539 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:43.325580 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:43.329145 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:29:43.329147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:29:43.329147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:29:43.329157 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:29:43.330954 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.331928 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.331930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.331930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.331993 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:29:43.332186 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:29:43.332187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:29:43.332187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:29:43.332325 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:29:43.332451 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:43.332695 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:29:43.332786 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:29:43.334549 24453 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [term 1 FOLLOWER]: Refusing update from remote peer 5932fe6911c24f3fab34cf33c2c40b9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:43.334880 24377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [term 1 FOLLOWER]: Refusing update from remote peer 5932fe6911c24f3fab34cf33c2c40b9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:29:43.335228 24486 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:43.335655 24486 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:29:43.338101 24513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5932fe6911c24f3fab34cf33c2c40b9b. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.338227 24513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.339564 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5932fe6911c24f3fab34cf33c2c40b9b. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.339679 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.342052 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.342180 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:43.342624 24499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.342731 24499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.342953 24537 mvcc.cc:205] Tried to move back new op lower bound from 7333274350934863872 to 7333274350817140736. Current Snapshot: MvccSnapshot[applied={T|T < 7333274350934863872}]
I20260925 15:29:43.344170 24540 mvcc.cc:205] Tried to move back new op lower bound from 7333274350934863872 to 7333274350817140736. Current Snapshot: MvccSnapshot[applied={T|T < 7333274350934863872}]
I20260925 15:29:43.344580 24525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.344691 24525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b [sys.catalog]: This master's current role is: LEADER
I20260925 15:29:43.345079 24513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5932fe6911c24f3fab34cf33c2c40b9b" member_type: VOTER last_known_addr { host: "127.1.163.126" port: 37549 } } peers { permanent_uuid: "4d73ea1afb464dbc9b2695b852796dcf" member_type: VOTER last_known_addr { host: "127.1.163.125" port: 43197 } } peers { permanent_uuid: "09ff55e4d00946d08401292b241fe0bf" member_type: VOTER last_known_addr { host: "127.1.163.124" port: 35309 } } }
I20260925 15:29:43.345175 24513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:29:43.345872 24538 mvcc.cc:205] Tried to move back new op lower bound from 7333274350934863872 to 7333274350817140736. Current Snapshot: MvccSnapshot[applied={T|T < 7333274350934863872}]
I20260925 15:29:43.346490 24532 catalog_manager.cc:1347] Generated new cluster ID: 7fd1ad8e5def45fbaa335660fa52b112
I20260925 15:29:43.346546 24532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:29:43.366020 24532 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:29:43.366964 24532 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:29:43.378091 24532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5932fe6911c24f3fab34cf33c2c40b9b: Generated new TSK 0
I20260925 15:29:43.378377 24532 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:29:43.399905 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.397956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.1.163.125:43197)
0925 15:29:43.398026 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.398028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.399115 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 15:29:43.399228 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.399229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.399242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.399274 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.399758 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55702: BlockingRecv error: recv got EOF from 127.0.0.1:55702 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 15:29:43.400717 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.398404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.1.163.124:35309)
0925 15:29:43.398458 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.398460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.400160 (+ 1700us) server_negotiation.cc:407] Connection header received
0925 15:29:43.400258 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.400259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.400270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.400300 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.400652 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56372: BlockingRecv error: recv got EOF from 127.0.0.1:56372 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:29:43.401502 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.398779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.1.163.126:37549)
0925 15:29:43.398831 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.398833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.400951 (+ 2118us) server_negotiation.cc:407] Connection header received
0925 15:29:43.401051 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.401052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.401064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.401093 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.401419 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39372: BlockingRecv error: recv got EOF from 127.0.0.1:39372 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:29:43.410128 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.408897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.1.163.125:43197)
0925 15:29:43.408954 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.408956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.409380 (+ 424us) server_negotiation.cc:407] Connection header received
0925 15:29:43.409506 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.409508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.409530 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.409563 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.410063 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55714: BlockingRecv error: recv got EOF from 127.0.0.1:55714 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 15:29:43.413861 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.412590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.1.163.125:43197)
0925 15:29:43.412632 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.412634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.413079 (+ 445us) server_negotiation.cc:407] Connection header received
0925 15:29:43.413200 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.413201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.413215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.413258 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.413787 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55726: BlockingRecv error: recv got EOF from 127.0.0.1:55726 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:29:43.418314 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.417373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.1.163.125:43197)
0925 15:29:43.417420 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.417423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.417707 (+ 284us) server_negotiation.cc:407] Connection header received
0925 15:29:43.417807 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.417808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.417834 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.417864 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.418254 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55732: BlockingRecv error: recv got EOF from 127.0.0.1:55732 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:29:43.424845 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.423729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.1.163.125:43197)
0925 15:29:43.423780 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.423782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.424113 (+ 331us) server_negotiation.cc:407] Connection header received
0925 15:29:43.424228 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.424229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.424242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.424274 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.424776 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55748: BlockingRecv error: recv got EOF from 127.0.0.1:55748 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 15:29:43.435315 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.434255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.1.163.125:43197)
0925 15:29:43.434306 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.434309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.434643 (+ 334us) server_negotiation.cc:407] Connection header received
0925 15:29:43.434759 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.434761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.434775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.434807 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.435251 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55750: BlockingRecv error: recv got EOF from 127.0.0.1:55750 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:29:43.453848 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.452795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.1.163.125:43197)
0925 15:29:43.452857 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.452859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.453189 (+ 330us) server_negotiation.cc:407] Connection header received
0925 15:29:43.453300 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.453301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.453313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.453347 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.453768 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55758: BlockingRecv error: recv got EOF from 127.0.0.1:55758 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:29:43.488248 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.487299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.1.163.125:43197)
0925 15:29:43.487350 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.487353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.487639 (+ 286us) server_negotiation.cc:407] Connection header received
0925 15:29:43.487744 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.487745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.487758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.487788 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.488187 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55764: BlockingRecv error: recv got EOF from 127.0.0.1:55764 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:29:43.554931 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.554046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.1.163.125:43197)
0925 15:29:43.554148 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.554151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.554280 (+ 129us) server_negotiation.cc:407] Connection header received
0925 15:29:43.554385 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.554387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.554400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.554479 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.554849 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55780: BlockingRecv error: recv got EOF from 127.0.0.1:55780 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:29:43.685706 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.684664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.1.163.125:43197)
0925 15:29:43.684762 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.684803 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.684909 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:29:43.684943 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.684944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.684956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.685018 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.685595 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55792: BlockingRecv error: recv got EOF from 127.0.0.1:55792 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:29:43.943951 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:29:43.943194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.1.163.125:43197)
0925 15:29:43.943343 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:43.943345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:43.943354 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:29:43.943392 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:43.943394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:43.943404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:43.943478 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:29:43.943902 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55796: BlockingRecv error: recv got EOF from 127.0.0.1:55796 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 15:29:44.298496 24510 catalog_manager.cc:1259] Loaded cluster ID: 7fd1ad8e5def45fbaa335660fa52b112
I20260925 15:29:44.298878 24510 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: loading cluster ID for follower catalog manager: success
I20260925 15:29:44.299847 24510 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: acquiring CA information for follower catalog manager: success
I20260925 15:29:44.300253 24510 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d73ea1afb464dbc9b2695b852796dcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:29:44.308780 24524 catalog_manager.cc:1259] Loaded cluster ID: 7fd1ad8e5def45fbaa335660fa52b112
I20260925 15:29:44.308952 24524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: loading cluster ID for follower catalog manager: success
I20260925 15:29:44.309998 24524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: acquiring CA information for follower catalog manager: success
I20260925 15:29:44.310385 24524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09ff55e4d00946d08401292b241fe0bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:29:44.461274 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:44.457544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.1.163.125:43197)
0925 15:29:44.457671 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:44.457673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:44.457702 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:29:44.457804 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:44.457804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:44.457903 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:44.457985 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:44.458387 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.458892 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.460100 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.460306 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.460374 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:44.460670 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:44.460804 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:44.460920 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:44.461099 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:29:44.461162 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260925 15:29:44.467586 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:44.464288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.1.163.124:35309)
0925 15:29:44.464457 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:44.464459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:44.464469 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:29:44.464599 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:44.464600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:44.464662 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:44.464751 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:44.464989 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.465654 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.466678 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.466888 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.466952 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:44.467158 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:44.467285 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:44.467430 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:44.467486 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:29:44.467512 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260925 15:29:44.474383 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:44.470775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.1.163.124:35309)
0925 15:29:44.470830 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:44.470832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:44.470935 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:29:44.471060 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:44.471061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:44.471073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:44.471105 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:44.471459 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.471995 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.473360 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.473752 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.473933 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:44.474091 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:44.474130 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:44.474205 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:44.474260 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:29:44.474283 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:29:44.474926 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:44.470743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.1.163.126:37549)
0925 15:29:44.470919 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:44.470921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:44.471003 (+ 82us) server_negotiation.cc:407] Connection header received
0925 15:29:44.471063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:44.471064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:44.471124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:44.471218 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:44.471568 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.472496 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.473594 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.473880 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.473941 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:44.474190 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:44.474321 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:44.474673 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:44.474781 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:29:44.474812 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260925 15:29:44.475044 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:29:44.470680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.1.163.125:43197)
0925 15:29:44.470775 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:29:44.470778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:29:44.470981 (+ 203us) server_negotiation.cc:407] Connection header received
0925 15:29:44.471094 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:29:44.471098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:29:44.471113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:29:44.471202 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:29:44.471590 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.472575 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.473743 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:29:44.474040 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:29:44.474104 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:29:44.474270 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:29:44.474304 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:29:44.474524 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:29:44.474906 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 15:29:44.474944 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:29:44.477650 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 24260
I20260925 15:29:44.483853 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 24335
I20260925 15:29:44.489725 1677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk86Bl6/build/release/bin/kudu with pid 24411
2026-09-25T15:29:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3062 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6150 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280459 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST